Model routing sends each request to the least expensive system capable of handling it well. Done carefully, it lowers latency and cost without making the product feel less capable.
Begin with a simple split
Route predictable work—classification, formatting, extraction, and short summaries—to a small, fast model. Send ambiguous, high-stakes, or multi-step requests to a frontier model. A rules-based first version is often easier to understand than a clever automatic router.
- Fast lane: short, structured, low-risk requests.
- Deep lane: long context, nuanced judgment, or complex reasoning.
- Escalation lane: retry with the stronger model when validation fails.
Add validation before complexity
Check whether the output follows the requested structure, contains required evidence, and passes domain-specific rules. Escalate failures. This feedback loop matters more than perfectly predicting difficulty upfront.
Measure accepted outcomes
Track latency, spend, escalation rate, and the percentage of results users accept without correction. The goal is not to route the most requests to the cheapest model; it is to minimize the cost of a successful task.
← Return to speed and cost