Self-Hosted LLM Inference Crossed a New Threshold, but Tokens/sec Is Still the Wrong Optimization

Self-hosted LLM inference is no longer a pure research exercise. In 2026 the combination of falling open-weight quality gaps, better serving stacks, and rising agentic token volumes has moved the economic decision into production engineering territory.

The claim that break-even has dropped to 1.2 million tokens per day does not hold up against careful TCO work. Multiple independent analyses place realistic crossover points substantially higher once GPU rental or amortization, power, monitoring, model updates, and engineer time are included. For mid-size and 70B-class models the numbers commonly land in the tens to hundreds of millions of tokens per day, or around 1.2 billion tokens per month for chat-style workloads when a full-time inference engineer is factored in. Against the cheapest hosted open-weight APIs the threshold moves even higher.

What has changed is the traffic shape. Agentic workflows multiply tokens through planning steps, tool calls, verification loops, and retries. A single user request can expand into dozens or hundreds of internal tokens. That multiplier accelerates the point at which fixed infrastructure cost undercuts usage-based pricing. At the same time, data-sovereignty and residency requirements (GDPR, sector-specific rules, customer contracts) frequently remove the pure cost option entirely. Sensitive traffic simply cannot leave the controlled perimeter.

The real engineering win is not another round of speculative decoding or FP8 tuning. It is knowing the exact volume and power envelope at which owning the stack becomes both cheaper and required.

Most teams still optimize the wrong primary metric. Tokens per second matters for interactive latency. Under power or data-center constraints, and especially on-prem, tokens-per-watt (or joules per token at the facility meter) is the binding limit. Longitudinal measurements show that software improvements alone have reduced energy per token on the same H100 generation by double-digit percentages between 2024 and 2025. Facility-level figures diverge sharply from chip TDP; only the former appears on the electricity bill.

Practical implications follow directly:

• Instrument blended cost (input + output + tool-generated tokens) under real agent traffic rather than synthetic benchmarks.
• Measure wall power and sustained tokens-per-watt, not just theoretical peak throughput.
• Treat residency requirements as a hard architectural constraint that can force ownership below the pure economic crossover.
• Keep hybrid routing: sensitive or high-value paths on the private stack, bulk or low-risk traffic on cheap APIs while volume grows.

Speculative decoding and quantization remain useful capacity levers. They do not replace the threshold calculation. Once utilization is high enough that the fixed cost is amortized, every additional agent step has near-zero marginal token cost and the compliance discussion changes character.

The decision is therefore operational, not theoretical. Track the actual daily token volume and power draw of your production agents. When the cloud line item exceeds the fully-loaded self-host floor, or when residency rules make the API path unavailable, the architecture choice is already made. The remaining work is execution: capacity planning, observability, and keeping the serving stack current. That is the technical deep dive that matters in 2026.