The orchestration problem is mostly solved. The routing problem isn't.
Okara's 4B tokens per day across multiple LLM providers on Vercel should be a canary in your coal mine. They're not wrestling with orchestration—LangGraph and Inngest have turned that into a commodity. They're fighting token routing, provider fallback, and cost control at volume. That's the real constraint.
Most agent teams still design as if there's one model behind the curtain. A single call to openai.ChatCompletion.create() with gpt-4o, everywhere. That works until you cross ~100M tokens/day. Then the bill arrives and the architecture has to change: multi-provider, per-sub-agent model selection, latency budgets that actually matter, fallback chains that survive 429s and 5xxs. By then you're refactoring at scale instead of planning for it on day one.
The cost delta between routing-aware and routing-naive stacks is often 3–5x. You can't solve that in the model layer alone.
Eight specialized sub-agents, eight different inference backends
A real agent system doesn't call one model. It calls many. A classifier/router on Claude Haiku or gpt-4o-mini (<300ms p95). Long-context retrieval synthesis on Gemini 2.0 or Claude with a 200k token window. Code generation on Claude Sonnet or GPT-4.1. Structured extraction on a fine-tuned open model via Fireworks or Together. A guardrail pass as its own cheap call. Embedding refresh as a separate cost line. Voice-tier sub-agents needing Groq or Cerebras for sub-100ms time-to-first-token.
Each of these has different latency expectations, context needs, and price points. Treating them all the same—routing them all to your "main" model—wastes tokens and burns budget. Right-sizing your agent's reasoning loop means picking the cheapest model that can do the work for each sub-agent, then building the router that enforces it.
Per-sub-agent latency SLAs are the contract your router enforces
Saying "my agent runs in 3 seconds" is meaningless. You need 2.5 seconds total: 200ms route + 400ms retrieve + 1.5s generate + 400ms tool. Each hop has its own budget and the router is what enforces it.
Track p50/p95/p99 per sub-agent, not per agent. Time-to-first-token and total completion are separate SLOs. When a provider hits latency tail (Anthropic's occasional 8-second responses, OpenAI rate-limit cascades), the router needs to cancel that call and fall back to a secondary provider within the remaining budget. That's AbortController semantics in the Vercel AI SDK, or equivalent timeout + cancellation in whatever gateway you build.
Fallback ladders work like this: primary provider → secondary provider → degraded model (cheaper, faster) → cached response. You need circuit breakers too—if a provider starts returning 429s or 5xxs, flip it off for 5–10 minutes and shed those requests to backups. Store that state in Redis, not memory, so all your gateway instances agree.
Provider fallback is a product decision, not an SRE detail
Anthropic's outages spike on the day they announce a new model. OpenAI's rate limits are regional and user-tier dependent. Google's quota exhaustion patterns are different again. Your fallback rules can't be generic; they have to fit the provider's actual failure modes.
What counts as "the same answer" across providers? If your sub-agent is generating JSON schema (structured output), fallback is safe—Claude and GPT-4 will agree on shape. If it's creative generation or tone-sensitive text, they won't. Build different fallback rules for deterministic vs. creative sub-agents.
Use idempotency keys + dead-letter queues to handle retried generations. Pair that with observability: Helicone, Langfuse, or homegrown OpenTelemetry spans per provider call. You need to see which providers are slow, which are failing, and what you're paying per sub-agent. The Vercel AI SDK's provider abstraction is a starting point, not the finish line—you'll add a thin gateway layer (your own, LiteLLM, Portkey, or OpenRouter) on top.
Cost routing: the 70/20/10 rule
We've started shipping with a simple heuristic: 70% of traffic to cheap/fast models at ~$0.25–1/M tokens (Claude Haiku, gpt-4o-mini, Llama 3.3 70B on Groq). 20% to mid-tier at ~$3–15/M tokens (Sonnet, GPT-4.1). 10% to frontier at ~$15–75/M tokens (Opus, o1). A difficulty classifier—itself a sub-agent call—determines which bucket a request belongs to. It pays for itself within ~50k requests.
Prompt caching cuts input cost 50–90% on repeated system prompts. Anthropic's 5-minute/1-hour windows and OpenAI's automatic caching both work. Layer in semantic caching (Redis or pgvector) for high-recurrence user prompts. Back-of-envelope: 4B tokens/day routed naively to frontier = ~$60M/year. Routed intelligently = ~$8–12M/year.
What to build on day one so you don't rebuild on day 400
Don't scatter SDK calls across your codebase. Build a thin gateway layer—your own or via LiteLLM/Portkey—and route everything through it. Log every call: sub-agent ID, model, provider, tokens in/out, cost, latency, retry count. Ship that to a warehouse (BigQuery, Snowflake) so your finance and engineering teams see the same numbers.
Feature-flag model swaps so routing changes don't require deploys. Build a synthetic eval set per sub-agent so you can A/B test providers without shipping regressions. Add a kill switch per provider for when (not if) one goes down for 4 hours.
If you're past 50M tokens/day and still calling SDKs directly, that's the moment to add a gateway. We've done this migration twice this year—reach out if you want a second set of eyes before the bill arrives.