RouteLLM
Open-source framework for cost-aware LLM routing
RouteLLM is profiled here as a Model Routing tool for engineering teams. Read about features, pricing, and how it compares to related options in the tools directory.
Description
RouteLLM is an open-source framework from LMSYS for routing requests between a stronger, costlier model and a cheaper, weaker one based on the difficulty of each query. Trained on preference data from Chatbot Arena, its routers predict when a request needs the stronger model and send everything else to the cheaper one, which the team showed can cut cost substantially while holding most of the quality. RouteLLM provides an OpenAI-compatible server so it drops into an existing client, and it ships routers and datasets for teams to evaluate and extend.
Key Capabilities:
Routers that send each query to a strong or weak model by difficulty
Preference-data training that learns routing from human comparisons
An OpenAI-compatible server for dropping routing into existing clients
Multiple router types including matrix factorization and classifiers
Generalization that lets trained routers work across model pairs
An evaluation framework for benchmarking routers on public datasets
Alternative tools
- Not Diamond
Intelligent router that picks the best model per prompt