LLM Routers Have Become a Service Category of Their Own
3 hours ago
- LLM routers automate selecting the most cost-effective model for each request, balancing quality and expense.
- The concept originated from IBM in 2021 and became a practical engineering pattern by 2024.
- Types include rule-based, semantic, predictive, cascading, and cost-based routers, often combined.
- Gateways like OpenRouter, LiteLLM, and Portkey focus on API unification and failover; smart routers like Martian and Not Diamond focus on per-request model selection.
- Neo-routers from Cursor, Ramp, and Meta's SwitchBoard aim to cut AI costs by routing simpler tasks to cheaper models.
- Routing is becoming a strategic layer between applications and model providers, creating tension as routers reduce spending while providers profit from premium models.