The HuggingFace framing gets utilization right and agents wrong
HuggingFace's recent piece on idle GPU capacity frames underutilized silicon as waste. That's correct for batch inference: if your GPU sits idle between requests, you're burning fixed costs on variable throughput. Traditional serving metrics like utilization % make sense there.
For agent teams, the story is different. GPU utilization % is a serving-side metric. For agents, the request isn't a single forward pass—it's a trajectory. A 6-step ReAct loop with 3 tool calls spends 60-80% of its wall-clock time outside the GPU entirely: making HTTP requests to tools, waiting for database retrievals, reassembling prompts. The GPU isn't idle between requests. It's idle between steps of a single request.
This is why a faster model—even a diffusion LLM that cuts decode time in half—barely improves your agent's latency. You saved 150ms on decoding. Your trajectory went from 8.5s to 8.35s. The bottleneck wasn't the GPU. It was the database request or the tool queue ahead of you.
GPU management for agents isn't a bin-packing problem. It's a latency-orchestration problem.
Where the latency tax actually lives in an agent run
Before you buy more capacity, know where the time goes. In the agent loops we've shipped, a typical trajectory breakdown looks like this: ~15% decode, ~25% retrieval (vector DB or Postgres round-trips), ~40% tool I/O (HTTP, auth, provider queues), ~20% orchestration overhead (prompt reassembly, re-tokenization, orchestration framework). Your mileage varies. Most teams measuring nothing have no visibility into their own bottleneck.
Tool-call round-trips are the enemy. A ReAct loop that calls a tool, waits for the response, then decides what to do next is serializing what could be parallel. Retrieval hops against pgvector or a managed vector DB add latency per step. Prompt reassembly and re-tokenization compound across steps. By the time you're on step 5 of 6, you've re-tokenized the same 4K-token system prompt four times.
There's also the split between time-to-first-token (TTFT) and total-trajectory-time. An agent cares about total time. You can't optimize TTFT if it means serializing tool calls.
The move is to instrument this before you optimize. Traces per step: which model ran, whether the cache hit, which tool was called, tokens in and out. Trajectory-level p50 and p95 wall-clock. Once you see the breakdown, the highest-leverage fixes become obvious.
Context caching beats bin-packing—if you actually measure hits
Anthropic's prompt caching and OpenAI's cached input pricing are real wins. A 6-step loop carries the same 4K-token system prompt and tool schema through all steps. You cache that prefix once, then read from cache 5 more times. That's 4,000 tokens you don't re-compute or re-pay for.
The challenge is keeping your prefix stable. If your system prompt drifts, your tool schema changes, or your few-shot examples shift, you cache-miss. A cache miss on step 4 of a 6-step loop is expensive. You re-pay for the whole prefix, wasting tokens and wall-clock time. Anthropic's 5-minute cache TTL fits agent loops fine—most run under that window—but prefix stability requires discipline. Store your system prompt and tool definitions in Redis or Postgres as the single source of truth. Construct the prefix byte-identical across all steps.
Then measure cache_read_input_tokens per step, not per request. Most teams enable caching and assume it works. We rarely see teams actually measuring hit rates. Target >70% on stable-prefix loops. If you're below that, you have a prefix drift problem, not a hardware problem.
Speculative batching and parallel tool calls: the two moves people skip
Parallel tool calling is the easiest win. OpenAI's tool_choice and Anthropic's parallel tool use let the model decide which tools to call together. Your job is to support it. Stop implementing serial ReAct. Parallel calls cut wall-clock time immediately.
Speculative decoding for the reasoning steps is the second move. vLLM and Text Generation Inference (TGI) both support draft models. Use a smaller model to speculate on reasoning tokens, then verify with your frontier model. The latency win compounds across a fleet.
The harder piece is batching across concurrent agent sessions at the token level, not the request level. If you have 50 agents running in parallel, they're all reassembling prompts, making tool calls, waiting for responses. Token batching across those 50 can hide retrieval latency. Idempotency keys and retry-with-backoff protect against double-charging downstream. Dead-letter queues catch tool failures that would otherwise stall a trajectory.
When a workflow—n8n, Temporal—beats an agent loop is exactly here. If your agent is mostly orchestrating parallel steps and waiting for results, you have a workflow, not an agent loop. The distinction is worth getting right; the maintenance cost is very different.
Offloading to cheaper endpoints is a latency decision, not just a cost one
Route classification, reformulation, and routing steps to Haiku, GPT-4o-mini, or a self-hosted 8B on vLLM. Reserve your frontier model for the one step that actually needs it. A Vercel AI Gateway or homegrown router lets you choose the model per step, not per request.
The reason to route isn't just $/token. It's p50 latency. A smaller model deployed on Google TPU v5e (Google Cloud's primary product revenue driver now) returns results faster than a frontier model in an API queue, even at lower throughput. Set per-step latency budgets: 200ms for routing, 800ms for retrieval synthesis, 3s for the reasoning step. Meet those budgets with the smallest model that can.
The economics compound across a fleet. Save 400ms per trajectory × 2 million trajectories per day = 800,000 GPU-seconds saved per day. That's equivalent to leaving dozens of A100s idle. Except you're not idling them. You're routing smarter.
What to instrument before you buy more GPUs
Spend an afternoon on this before you negotiate reserved capacity. Add per-step spans in OpenTelemetry with model, cache_hit, tool_name, tokens_in and tokens_out. Trajectory-level p50 and p95 wall-clock, not request latency. Cache hit rate as a first-class SLO.
Track tool-call error rate and dead-letter queue depth. Cost-per-successful-trajectory, not cost-per-token. When you can answer "on a 4-step loop, what's my p95 wall-clock?" and "how many tokens did step 1 read from cache vs. step 5?", you'll know whether your problem is silicon or orchestration.
Teams stuck above 8s p95 on a 4-step loop usually have a caching or serialization bug, not a hardware problem. That's when to reach out. Before that, fix prefix stability, enable parallel tool calls, and route small steps to small models.
The next move: compare cache_read_input_tokens across steps 1 and N of the same trajectory. That single graph tells you whether your problem is physics or plumbing.