The M365 Copilot default just repriced your agent stack
Microsoft shipping GPT-5.6 as the default in Microsoft 365 Copilot signals a hard shift in how enterprises meter agent costs. It's not tokens anymore. It's task completion quality — per-seat pricing at the enterprise level, but under the hood, per-successful-action at the model layer.
That reprices every agent stack we've shipped. The old playbook — swap GPT-4o for a $3/M input model to cut spend — breaks when the stronger model finishes in one loop and the cheaper one needs three. A frontier model paying $15/M input but landing on the first try beats a budget variant paying $3/M and looping twice, taking tool calls, hitting timeouts, triggering retries. The unit you're actually paying for isn't tokens. It's working actions that complete.
Cost-per-token is the wrong denominator for agents
Here's a worked example. A task requires three sequential tool calls: validate data, call an external API, write to a database. Your cheap model loops four times, generating redundant API calls and triggering retries on 429s. Your frontier model lands in one loop, one API call, one write.
Cheap model math: 3,500 tokens × 4 loops = 14,000 tokens, plus 12 tool calls (3 per loop × 4), plus 2 retries on timeouts, plus 15 minutes of human review because the output needs sanity-checking. Frontier model math: 2,800 tokens × 1 loop = 2,800 tokens, plus 3 tool calls, plus zero retries, plus two minutes of review. Per-token, the cheap model looks 80% cheaper. Per-successful-action, the frontier model is two-thirds the cost.
We've built this wrong. Most teams measure token burn in isolation. When you right-size your agent's reasoning model, the real denominator is: (input tokens + output tokens + tool-call latency cost + retries + downstream API spend + human-in-the-loop escalations) / number of completed tasks. That changes everything.
Anthropic Claude and OpenAI's GPT-5 tier cost differently, sure. But at the agent layer, the trade-off is loop depth and retry count. Count those first before you swap models.
Your retry logic is where the savings actually hide
Here's what we now ask before any model swap: What's your current retries-per-successful-action? Most teams don't know.
Retries live in three places. Model retries (bad JSON, refusal, hitting the context window) trigger another generation pass — you're paying tokens again. Infrastructure retries (429 from Anthropic or OpenAI, 5xx timeout) add latency and tool-call overhead without model cost but block the thread. Dead-letter queues in Redis or SQS catch terminal failures, but infinite retry loops are where budgets die.
Wire exponential backoff with jitter on 429s and 529s. Instrument idempotency keys on any tool call that mutates state — Stripe, internal APIs, database writes. One retry that double-writes costs human time, data-fixing sprints, and compliance audits. Cap loop depth with max_iterations as a cost ceiling, not a correctness knob. And log attempt count per task_id so you can slice retries-per-success by model, time-of-day, and task type.
The team that debugs RAG at production scale already knows this: observability at the task level, not the request level, is where you find the real spend leaks.
You can't optimize what you don't trace end-to-end
Cost-per-successful-action requires a Postgres schema and observability instrumentation most teams skip. You need task_id, model, input_tokens, output_tokens, tool_calls, retries, success_bool, wall_ms. Use OpenTelemetry spans tagged with task_id, model, attempt_n, and terminal_state. Route those spans to Langfuse or your APM so you can slice per-task cost rollups.
Define 'success' explicitly. Is it an eval rubric score, a downstream API response code, or user confirmation? Without that signal, your cost dashboard lies. P50 latency lies worse.
The engineering pattern that works: one span per task, child spans for each tool call and model invocation, and a success_bool signal at the task level. When you roll this up across your production fleet, you'll see which model, which task type, which time-of-day has the worst completion-to-cost ratio. That's your routing lever.
Route by difficulty, not by default
Once you measure honestly, the pattern emerges: tiered routing. Hard tasks go to the frontier model. Easy ones go to the cheap variant. The routing rule itself gets observed and tuned like any other system.
Drop a classifier or heuristic router in front of your agent — Vercel AI Gateway rules, custom middleware, or even simple heuristics like token count on the input. Escalation patterns work too: start on the cheap model, promote to GPT-5.6 or Claude Opus on retry. Circuit-breaker by budget per task_id stops runaway spend mid-flight.
Sometimes the agent itself is the wrong choice. If you can describe the work as a flowchart, skip the agent and ship a deterministic workflow. Measure that decision the same way — cost-per-successful-action lets you compare agent and workflow directly.
What we now ask before recommending a model swap
On client engagements, we use a checklist. Apply it Monday morning.
- Do you log attempt count and terminal_state per task_id?
- What's your current retries-per-success and P95 loop depth?
- Are tool calls idempotent, or does a retry double-charge or double-write?
- Have you priced human review time into the denominator?
- Is there a routing layer, or is every task paying frontier prices?
- Can you compute cost-per-successful-action for your current model and one tier up?
If you don't have answers to the first three, swap models only for observability gains. The cost savings will reveal themselves later, but only if you wire the measurement layer first.
Instrument one production agent this week with task_id, attempts, tokens, tool calls, and terminal state. Compute cost-per-successful-action across your current model and one tier up. If the numbers surprise you, talk to us about routing the hard tasks up.