Across layers · read in stack order
Self-hosting vs routing vs measuring: cutting inference cost
Inference cost can be attacked at the substrate, in the router, or by first finding out where it goes. Teams usually start with the most expensive of the three — standing up their own GPUs — when the cheapest would have told them it was unnecessary.
Short answer
Measure first. Per-request cost attribution is the cheapest of the three and the only one with no quality risk, and it often shows a few features or prompts dominating spend — which a shorter prompt can fix without new infrastructure.
By Lattice editorial · Licence and cost facts verified · method
- vLLMLayer 01 · Band I ComputePay for GPU-hours instead of tokens.
- RouteLLMLayer 02 · Band I ComputeSend easy queries to a cheaper model.
- HeliconeLayer 09 · Band III ControlFind out which requests cost the most.
| Dimension | vLLM | RouteLLM | Helicone |
|---|---|---|---|
| Lever | Own the serving, pay for hardware | Match query difficulty to model size | Attribute spend per request, user and feature |
| Saves money when | Traffic is high and steady enough to keep GPUs busy | A meaningful share of queries are easy | Spend is concentrated somewhere you have not looked |
| Loses money when | Utilisation is low — idle GPUs still bill by the hour | The router misjudges hard queries and answers degrade | Never on its own — it saves nothing until you act on it |
| Quality risk | None if you serve the same model; real if you downsize to fit | Direct — this is a quality-for-cost trade by design | None |
| Operational cost | A serving fleet, on-call and capacity planning | A router to calibrate, plus an eval to trust it | A proxy hop or an SDK wrapper |
| Adopt first when | You know your utilisation and it is high | You can measure the quality you are trading away | You cannot yet say which feature costs the most |
Scroll the table sideways to see every tool.
What this table is not. These are editorial judgements, not benchmarks: Lattice has not run these tools head to head, and no cell uses GitHub stars or vendor benchmark claims as evidence. Licence and cost facts are re-checked on a schedule the build enforces — the receipt is public.
The order
Which should you adopt first: vLLM, RouteLLM or Helicone?
Measure first. Per-request cost attribution is the cheapest of the three and the only one with no quality risk, and it often shows a few features or prompts dominating spend — which a shorter prompt can fix without new infrastructure. Route next, but only once an eval can tell you what the cheaper model loses. Self-host last, when measured and sustained utilisation makes GPU-hours cheaper than tokens; below that line it raises the bill.
Rules of thumb
- Cost per token is not cost per answer. A cheaper model that needs two retries is the expensive one.
- Compare self-hosting against the price you actually pay, at the utilisation you actually have — not at peak.
- A router without an eval is a cost cut with an unknown quality bill attached.
Quick answers
- When should you use vLLM, and when should you skip it?
- Use vLLM when: general GPU serving under bursty, high-concurrency traffic. Skip it when: traffic is steady and low-concurrency, so batches never fill.
- When should you use RouteLLM, and when should you skip it?
- Use RouteLLM when: cutting cost by matching query difficulty to model size, with a learned router. Skip it when: you cannot evaluate the quality you would be trading away.
- When should you use Helicone, and when should you skip it?
- Use Helicone when: gateway-level observability with per-request cost and latency. Skip it when: you need eval primitives more than traffic logs.