How We Cut LLM Inference Cost ~20% on a Voice Platform That Scaled Past 1,500 Paying Customers

When you run real-time voice agents at scale, silence is a bug and cost is product debt.

At Callin.io we built and operated a production voice platform that grew past 1,500 paying customers. The work included medical and real-estate enterprise accounts. As Technical Lead I owned the architecture, vendor spend, model routing, telephony failover, and the billing system itself. One of the clearest results was a roughly 20% reduction in LLM inference cost while keeping typical voice time-to-first-token low enough that callers stayed on the line, and while making billing disputes rare enough that they stopped being a meaningful support load.

This is not a theoretical cost-optimization checklist. It is the set of decisions that actually moved the numbers.

The problem we had to solve

Three pressures hit at the same time:

• Every conversational turn is a race. If the model thinks too long, the caller hangs up. Latency is the product.
• Using one frontier model for every utterance (greetings, confirmations, simple FAQs, and hard reasoning) burned margin on the easy turns.
• Minute-based billing with imprecise tracking created support tickets and eroded trust with the accounts that mattered most.

We needed the system to be fast enough that humans stayed on the line, cheap enough that the unit economics worked at volume, and boringly correct on the invoice so customers stopped arguing about charges.

Complexity-aware model routing

The single highest-leverage change was stopping the practice of sending every turn to the same large model.

We classified each turn by intent and complexity:

• Greeting / confirmation / simple FAQ → smallest model that could finish the job reliably
• Scheduling, basic information retrieval → mid-tier model
• Objection handling, multi-step reasoning, edge cases → frontier model only when required

The classifier itself was deliberately lightweight. The goal was not perfect intent detection. The goal was to keep the expensive models off the critical path for the majority of turns. Large models were reserved for the moments where the extra intelligence actually changed the outcome of the call.

This single change produced the bulk of the ~20% LLM cost reduction. It also improved tail latency because the smaller models returned first tokens faster on the high-frequency paths.

Semantic cache + concurrent prompts

Even with routing in place, repeated or near-repeated intents still hit the model. We added a semantic cache for high-frequency patterns and fired retrieval and response scaffolds in parallel so that time-to-first-token dropped before the caller registered silence.

The cache was not a blunt exact-match store. It was keyed on semantic similarity of the current turn against previously successful responses in the same context. When a hit occurred, we could start speaking almost immediately. When it missed, the concurrent scaffolding meant the model still had a head start.

In practice this combination (routing + cache + parallel work) kept the typical voice TTFT in a range where hang-ups became rare enough that we stopped treating latency as a crisis metric and started treating it as a solved constraint.

Dual-carrier telephony and ownership of the full path

Cost and latency are not only model problems. They are also carrier and orchestration problems.

We ran dual-carrier telephony (Twilio + Telnyx) with SIP fallback. Failover had to happen without dropping the call and while audio continued streaming over WebSockets under load. Carrier choice and failover logic sat on the same turn clock as the model routing decisions. Owning the full path from PSTN/WebRTC through the orchestrator to the voice out (ElevenLabs or Cartesia) meant we could optimize the entire chain instead of blaming individual vendors.

Billing as a product surface

The final piece was treating billing itself as a product surface rather than an afterthought.

We built minute tracking with rollover ledgers precise enough that disputes became rare. Stripe subscriptions were wired to actual usage, not estimates or rounded buckets. When a customer looked at an invoice they could see the same numbers we saw internally. The result was near-zero billing disputes after launch, even as volume grew into enterprise healthcare and real-estate accounts.

If customers argue about invoices, the product is unfinished. Making the ledger boringly correct removed an entire class of support work and increased trust with the accounts that paid the most.

What the numbers looked like

• Platform grew past 1,500 paying customers
• LLM inference cost reduced by approximately 20% through complexity-aware routing and caching
• Typical voice TTFT held low enough that callers remained on the line
• Billing disputes dropped to near zero after the ledger work shipped
• The same multi-LLM architecture later supported adjacent products (CondoMail, Realead) without starting from scratch

Practical takeaways

1. Route by complexity. A confirmation does not deserve a frontier model. Your CFO will notice. So will your p95.
2. Treat silence as a bug. Every architectural choice (cache, parallel prompts, carrier failover) exists to keep the human from hanging up.
3. Own the stack’s P&L. Architecture without vendor-spend ownership is theater. Know what the infra bill was last Tuesday.
4. Make billing correct enough that it disappears as a support topic. Usage ledgers should be boring.
5. Measure the full turn, not just the model. Latency and cost live across telephony, orchestration, retrieval, and synthesis.

The work was never only about making the models cheaper. It was about making the entire system fast enough, cheap enough, and trustworthy enough that 1,500 companies would keep paying for it, and that the team could still sleep at night.

If you are building or operating voice agents at any meaningful volume, the questions worth asking are not only “which model is best?” but “which turns actually need the expensive model?” and “will the invoice match what the customer experienced?”

Those two questions, answered rigorously, produced the 20% cost reduction and the near-zero dispute rate more reliably than any single model swap ever did.