One-Command vLLM Is Table Stakes, Not a Strategy
HuggingFace Jobs made self-hosted inference friction-free. You point at a model card, select an H100, pick a quantization, and hit deploy. Sixteen minutes later you have an OpenAI-compatible endpoint serving Llama 3.1 70B or Mistral Large. That's real. vLLM's continuous batching and PagedAttention have made the serving layer genuinely good—no more building custom inference stacks from NVIDIA container artifacts and debugging OOM errors in production.
But the moment that button works, the hard decision was already made. Deployment friction was never the constraint. The constraint is whether you should self-host at all.
The Break-Even Is a Token-Volume Calculation, Not a Vibe
Here's the math we run with clients. An on-demand H100 on HF Jobs runs $2–4 per hour depending on region and commitment. A single H100 can sustainably push roughly 15,000–20,000 tokens per second (input + output) with reasonable latency, assuming you're actually batching and not letting the GPU sit half-empty.
GPT-4o-mini costs $0.075 per 1M input tokens and $0.30 per 1M output tokens. Claude 3.5 Haiku runs $0.80/$4.00 per 1M. If your workload is mostly reasoning on small context, you're looking at an effective blended rate around $0.15–0.25 per 1K tokens. That's $1.50–2.50 per billion tokens. A self-hosted H100 at $3/hr can serve about 54–72M tokens per hour, which means you're paying roughly $0.04–0.06 per 1K tokens on the GPU cost alone—if you're at full utilization.
The break-even is sharp. If you're running under 500M tokens per day, staying on the API is cheaper, simpler, and you don't own the on-call pager. At 2B tokens per day with sustained traffic, self-hosting the right open model (Llama 3.1 70B, Qwen2.5 72B) starts to win on raw cost. But that assumes your traffic is predictable, your utilization actually stays high, and you're not burning engineering hours keeping the thing running.
Most teams underestimate utilization. A GPU sitting at 40% utilization because traffic arrives in spikes still costs $3/hr. That's $21.6K per month for 40% of an H100. The API absorbs your spikes. You inherit them.
Latency: The p99 Story the Marketing Page Won't Tell You
vLLM's continuous batching is real, and it shrinks time-to-first-token (TTFT) at median load. An API provider's inference fleet is sized for p99 latency across thousands of concurrent requests. Your on-demand H100 in HF Jobs is one endpoint. When you get burst traffic—your agent spawns six parallel tool calls, or a sales team suddenly runs 50 outbound calls—your GPU queue fills and p99 TTFT climbs to 500–800ms, where the API's stays at 150–200ms.
For agents, this is a multiplier. A single tool call at 200ms tail latency is annoying. Six tool calls in a loop, each hitting an agent's reasoning model, means 1.2 seconds of latency that the user feels. Self-host that reasoning layer and your p99 doubles on bad days. That's the difference between a responsive agent and one that feels sluggish.
If your traffic shape is genuinely flat and predictable—a batch job that runs at 11pm, stable throughput with no bursts—self-hosting's latency story is fine. If it's a user-facing agent or a product that sees traffic spikes, staying on the API buys you latency tail coverage that an H100 won't give you.
Failure Modes You Inherit the Moment You Self-Host
An API call failing and retrying is a solved problem. You own exponential backoff, idempotency keys, dead-letter queues. HuggingFace Jobs handles the container and the GPU driver, but they don't handle your request routing or your monitoring.
The moment you have a self-hosted vLLM endpoint, you own: OOM errors on long-context inputs (no automatic fallback to a smaller context window); CUDA driver crashes (they happen; we've shipped both); model weight versioning (which HF revision are you running? can you rollback if a quantization breaks on Tuesday?); observability (Prometheus scraping your vLLM metrics, request logs in Loki or CloudWatch, token accounting so you know why your utilization is lower than planned); on-call rotations (someone pages at 2am when the endpoint goes silent); redundancy (one endpoint is a single point of failure—you'll eventually need two H100s, which erases the cost advantage).
An API-native system is simpler. It's not zero-operational-surface, but it's smaller.
Vendor Lock Cuts Both Ways
The "lock-in to Anthropic/OpenAI" argument gets trotted out whenever self-hosting comes up. It's real—you'll tune your prompts to a model's quirks and switching costs are genuine. But self-hosting isn't lock-in-free. It locks you to H100 availability (the supply chain for enterprise GPUs is actually tight), to HF Jobs as a runtime, and to vLLM's breaking changes (they happen; the API compatibility layer isn't frozen).
The best pattern we've seen is hybrid. Use an OpenAI-compatible gateway (vLLM exposes one natively) in front of your self-hosted endpoint. Route your stable, high-volume, predictable traffic there. Route frontier reasoning—the 5% of queries that need GPT-4o or Claude for domain reasoning you can't fit into a 70B model—to the API. That way you're not locked into either one. The cost is a routing layer; the payoff is optionality.
If you're early-stage and still A/B-testing which model works for your domain, stay on the API. Switching models week-to-week on self-hosted infrastructure is noise on top of your signal.
A Decision Rule We Actually Use with Clients
Here's the checklist we bring to planning meetings. First: what's your traffic shape? Is it predictable and flat (batch job, internal tool), or spiky and user-dependent (chatbot, agent)? Predictable traffic justifies self-host earlier.
Second: are you still iterating on models and prompts? If you're A/B-testing weekly, stay on the API. The friction of redeploying a container and re-tuning quantization on HF Jobs is real. Once you've settled on a model, stability favors self-host.
Third: compliance or data residency. If your data can't leave your region or your security model requires it, self-host. If not, it's a cost conversation.
Fourth: if you clear the bar for self-hosting, don't flip a switch. Deploy vLLM in parallel to your API for two weeks. Shadow-deploy: send production traffic to both, log both responses, compare latencies and errors. When the shadow deployment stabilizes, cut over the traffic. If something breaks, you have a kill switch.
We've watched teams skip that step. They don't learn from it, but we do. The two-week shadow is insurance.
Stay on the API until you're burning more than $5K–10K per month on a single stable model with genuinely predictable traffic. Once you clear that bar, run a vLLM shadow deployment for two weeks before you commit. If you want us to run that math with you or build that infrastructure, we're here.