Lattice
Skip to content
← Fix a symptom

Band I · Compute · “too slow, too expensive”

Why is my LLM bill so high?

Short answer

LLM spend is rarely spread evenly: a few features, long prompts or silent retries usually dominate it. Attribute cost per request and feature first. Then shorten prompts and outputs, cache repeats, and route easy queries to cheaper models — with an eval confirming quality held. Self-host only when sustained utilisation makes GPU-hours cheaper than tokens.

By Lattice editorial · Tool facts verified · method

Check in this order

  1. 01

    Attribute spend per request, user and feature

    Layer 09 · Evaluation & Observability

    It is the only cut with no quality risk, and it tells you which of the others is worth doing.

    • Helicone
      Use when
      Gateway-level observability with per-request cost and latency.
      Skip when
      You need eval primitives more than traffic logs.
    • Langfuse
      Use when
      Self-hostable tracing, prompts and evals in one place.
      Skip when
      You want a fully managed product with a support contract.
  2. 02

    Shrink the prompt and the context you send

    Layer 08 · Prompt Engineering

    Input tokens are paid on every call. Context stuffed in just in case is the most common silent cost.

    • DSPy
      Use when
      Compiling prompts into optimised programs from labelled examples.
      Skip when
      You have no examples and no way to score them.
  3. 03

    Cache repeats and enforce budgets at the gateway

    Layer 02 · Routing & Gateways

    A budget is only enforceable where spend is attributed, and a cache hit is a call you do not pay for.

    • LiteLLM
      Use when
      The widest provider coverage, deployed the least committal way.
      Skip when
      You want routing policy to live in a SaaS console.
    • Portkey
      Use when
      A managed gateway with guardrails and analytics already bundled.
      Skip when
      Prompts cannot leave your infrastructure.
  4. 04

    Route easy queries to a cheaper model

    Layer 02 · Routing & Gateways

    A real saving when many queries are easy — but it trades quality for cost by design, so it needs an eval first.

    • RouteLLM
      Use when
      Cutting cost by matching query difficulty to model size, with a learned router.
      Skip when
      You cannot evaluate the quality you would be trading away.
    • Not Diamond
      Use when
      Routing and prompt optimisation tuned on your own traffic.
      Skip when
      Self-hosting or auditability is a requirement.
  5. 05

    Self-host, if utilisation is high and steady

    Layer 01 · Inference & Serving

    GPU-hours beat per-token pricing only when the GPUs stay busy. Below that line it raises the bill.

    • vLLM
      Use when
      General GPU serving under bursty, high-concurrency traffic.
      Skip when
      Traffic is steady and low-concurrency, so batches never fill.

Looks like a fix, is not

  • Negotiating a volume discount before you know which feature spends the money.
  • Routing to a cheaper model with no eval to say what it loses.
  • Self-hosting at low utilisation — idle GPUs still bill by the hour.

Quick answers

What should I check first?
Attribute spend per request, user and feature. It is the only cut with no quality risk, and it tells you which of the others is worth doing.
What looks like a fix but is not?
Negotiating a volume discount before you know which feature spends the money. Routing to a cheaper model with no eval to say what it loses. Self-hosting at low utilisation — idle GPUs still bill by the hour.