What Routing Rules Actually Move Out of Your Codebase
Vercel AI Gateway's routing rules do one thing decisively: they move model failover and provider swaps from deployment-time decisions to runtime configuration. Before this layer existed, your options were hardcoded provider chains in application code or a feature flag in Postgres or LaunchDarkly. Now you write a rule as YAML or JSON at the gateway, define a primary provider (Anthropic) and fallback chains (OpenAI, then Bedrock), and propagate that decision in seconds instead of a 30-90 minute deploy cycle.
That's real. An Anthropic outage at 3am used to mean a PR, code review, merge, build, deploy, wait for edge caches to refresh. Now it's a config change and you're on Bedrock Claude in 10-15 seconds. The gateway watches for specific failure modes—429 rate limits, 5xx errors, timeouts beyond your SLA—and switches providers automatically without touching your application code.
What stays in code: prompts, tool schemas, retry logic, idempotency keys, and the retry strategy itself. The gateway handles provider-level failover. It does not handle the fact that Claude 3.5 Sonnet and GPT-4o have completely different tool-calling semantics.
The Half It Doesn't Solve: Behavioral Drift
Anthropic uses native tool_use blocks. OpenAI uses function calling in the message stream. You can swap providers at the gateway and fail silently for weeks because your evals pass, your logs look clean, and then users start filing tickets because the agent is refusing tool calls it should accept, or tool calls are malformed in ways you haven't instrumented.
Tool-calling format differences are just the beginning. JSON mode in OpenAI works differently than Anthropic's structured output constraints. Refusal rates shift—Claude 3.5 Sonnet refuses certain classes of requests more aggressively than GPT-4o. XML tagging expectations differ. Prompt engineering that works on one provider requires variants for another, but your routing rule doesn't know which model is handling the request at execution time, so you can't branch the prompt.
Silent regressions are the real operational cost. You swap Claude for GPT-4o at the gateway because of availability, and 72 hours later your team is debugging why tool planning accuracy dropped from 94% to 87%. You have no direct trace from the incident to the routing change. This is why any routing rule change needs an eval harness that gates the deployment and exposes behavioral deltas before they reach production. A test suite that runs your agent's decision loop against a golden dataset of 200-500 real requests, comparing output shape and tool-call correctness across providers.
The pattern we've seen work: one eval per provider, not one eval with N backends. You measure tool-call precision, refusal false-positive rate, and JSON schema compliance separately for each provider. You gate any routing rule change—even a fallback chain reorder—on eval results. This is expensive in the moment. It's free compared to 48 hours of production debugging.
Cost and Latency Routing Is an Application Concern, Not a Gateway One
Routing rules excel at availability failover. They fail at cost and latency optimization because the gateway can't see the semantic intent of the request.
A real pattern: your agent receives a user query, classifies it in 200ms on GPT-4o mini ($0.25/M tokens), then routes 5% of complex planning tasks to Claude 3.5 Sonnet ($15/M tokens). The cheap model handles 95% of traffic, the frontier model handles the hard 5%. You can't express this at the gateway. The gateway sees "request came in" and "model X available"—it doesn't see "this is a summarization call" vs. "this is multi-step planning."
This logic belongs in your agent loop, instrumented with token accounting per-tenant and per-step. Semantic caching—checking Redis or Upstash before calling the gateway at all—handles another chunk. But the real win is classifying the request in code, where you have context. Then your routing rule only needs to handle "primary unavailable, use fallback," not "route by cost and latency."
Overprovisioning frontier models is a habit we've had to break. Teams ship GPT-4o or Claude 3.5 Sonnet for every request, then wonder why their token spend is 10x the competitor. Right-sizing means classify, then route. The gateway handles failover. The agent handles optimization.
When Routing Rules Are Premature Infrastructure
Skip routing rules if you're under 1M tokens/day with a single provider. You don't have the operational churn to justify the complexity. A single provider with a hardcoded fallback—or a feature flag in Postgres—covers 80% of the value and adds zero operational surface.
You're not ready if you don't have an on-call rotation yet. Routing rules buy you incident response speed. If you're the only engineer and you're asleep, it doesn't matter that failover is 10 seconds instead of 90. You still have to debug it at 6am.
The real trap is shipping multi-provider routing before you have evals, observability, or a golden dataset. We've watched teams deploy Vercel AI Gateway routing rules, then spend weeks chasing behavioral regressions they didn't instrument for. A feature flag in LaunchDarkly with a canary toggle—promoting to 10% of traffic manually—solves this for months. Swap it if you get two provider incidents in a quarter that hurt users, not before.
When They Genuinely Pay Off
Multi-region, multi-tenant agents pushing 100M+ tokens/day. Regulated workloads needing region-pinned providers (EU-only via Bedrock Frankfurt). Teams with an on-call rotation and incident response procedures.
If you're here, you need routing rules paired with:
- Distributed tracing (OpenTelemetry, Langfuse, Helicone) so you can attribute behavioral regressions to a specific routing change. A trace ID that travels from user request through the gateway to the provider, tagged with which model handled it.
- Dead-letter queues for failed tool calls, not just failed completions. A tool call that errors silently is worse than a model timeout because you won't see it in your standard observability.
- Per-model canary at 5% traffic before promoting a rule. Ship the new provider or fallback chain to 5% of production, measure latency and error rates for 4-8 hours, then promote or rollback.
- A rollback playbook that's rehearsed, not theoretical. Document exactly what happens when you disable a provider at 2am, which traces to watch, and how to validate the rollback worked.
At token scale, orchestration is solved—routing is the hard problem. But it's solved by pairing multiple layers.
The Stack We'd Ship Around It
Layer 1: In-application classifier and semantic cache. Classify intent on a cheap model, check Redis before hitting the gateway at all. Token spend drops 40-60% and your latency budget is tighter.
Layer 2: Vercel AI Gateway for provider failover and rate-limit smoothing. Primary provider + fallback chain. You own the decision to swap, the gateway owns the execution.
Layer 3: Eval gate in CI. Every prompt change, every routing rule change, every model swap runs through a test suite that compares tool-call correctness and refusal rates across providers. Don't ship if eval doesn't pass.
Layer 4: Per-decision safety filtering. Not one gateway-level filter that applies to everything. Different requests have different safety profiles. Route the heavy ones through additional checks.
Layer 5: Observability with trace IDs propagated from user request → gateway → provider. Cost dashboards by feature, not by provider. Attribution matters more than raw spend.
If you're staring at a routing rule config wondering whether it's solving your real problem or hiding it, let's talk. We've shipped this layer for teams at both ends of the token-volume spectrum, and the difference between "we have routing rules" and "we have routing rules that don't break things" is usually three weeks of instrumentation you didn't expect.