Writing ·
Multi-model routing cost controls for AI agents in 2026
Multi-model routing controls cost by making the harness choose the cheapest reliable lane for each agent step, then measuring accepted output instead of raw call spend.
By Youssef Hemimy · cost controls · agent reliability · routing
Multi-model routing controls cost by making the harness choose the cheapest reliable lane for each agent step. The useful unit is not token spend in isolation; it is accepted output after retries, fallbacks, latency, and human review. Treat routing as an operations policy, not a prompt trick.
Classify
Control plane
Route
Feedback loop
Route decisions stay adjustable only when outcomes, cost, latency, and escalation reasons are logged with the final result.
Route decisions need a control plane
The routing decision should be visible, repeatable, and reviewable. A production harness can classify the task, apply policy, check budget, choose a lane, and log the reason. A prompt can only ask the model to be economical after the expensive decision has already happened.
Optimize for accepted output
A cheaper model call can become more expensive when it causes retries, escalation, rework, or missed latency targets. The practical metric is cost per accepted output: the total route cost divided by the work that survived review and reached the user.
| Cost view | What it misses | Better operating view |
|---|---|---|
| Cost per call | Retries and failed attempts | Cost per accepted output |
| Average latency | Tail latency on hard tasks | p95 latency by route |
| Fallback count | Whether the fallback fixed the task | Fallback acceptance rate |
| Human review volume | Why review was needed | Escalation reason by route |
Use three lanes before adding more
A simple routing policy is easier to debug than a maze of model choices. Start with a fast lane for low-risk work, a deep lane for complex or high-context work, and a fallback lane for recovery. Add more lanes only when logs show a repeated task shape that does not fit the existing policy.
- Fast lane: deterministic transforms, summaries, classification, and other low-risk work with clear acceptance checks.
- Deep lane: ambiguous planning, high-context synthesis, policy-heavy work, and tasks that affect customer-visible decisions.
- Fallback lane: retry after a route-specific failure, with the original route decision preserved in the trace.
Budgets must stop work, not just report it
A dashboard that reports spend after the fact is accounting. A routing budget is an active guard: it can downshift a low-risk task, require review for a high-risk one, or pause a workflow before a runaway loop becomes a bill and an incident.
What to instrument first
The first useful routing dashboard is small. Track the selected route, route reason, accepted-or-rejected outcome, retry count, fallback use, latency, cost, and escalation reason. That gives reviewers enough evidence to change policy without guessing.
Bonfire builds this kind of harness work into agent infrastructure engagements and uses it to decide which automation should be hardened, rewritten, or left human-led. The routing system is not valuable because it is clever; it is valuable because it makes cost decisions observable.
FAQ
Should every agent workflow use multiple models?
No. Multi-model routing pays for itself when tasks vary by risk, context size, latency tolerance, or review requirements. If every task has the same shape, a single well-instrumented lane is usually easier to operate.
Where should routing logic live?
Put routing in the harness, not in prompts scattered across agents. The harness can log the decision, enforce budgets, apply fallback policy, and compare outcomes over time.
What is the biggest failure mode?
Routing on price alone. The cheaper lane is not cheaper if it creates more retries, escalations, or human review work. Track cost against accepted outputs, not against raw model calls.
How does this connect to AgentOps Hardening?
Routing is one control inside AgentOps Hardening. It belongs beside retry policy, idempotent tools, checkpoints, and observability because a route decision changes reliability behavior, not just spend.
Building something that has to hold up?
We do this work for teams — agent reliability hardening, custom MCP servers, and full-stack AI systems built to survive production.