Why is my LLM app slow?

Check in this order

1. 01

   Trace one slow request end to end

   Layer 09 · Evaluation & Observability

   Latency hides in sequential tool calls, retries and retrieval as often as in generation. You cannot shorten a span you have not seen.

   • Langfuse

     Use when

     Self-hostable tracing, prompts and evals in one place.

     Skip when

     You want a fully managed product with a support contract.

   • Arize Phoenix

     Use when

     OpenTelemetry-native evaluation you can run yourself.

     Skip when

     You want a vendor support contract behind it.

   • OpenTelemetry

     Use when

     A vendor-neutral standard for emitting traces and metrics.

     Skip when

     You want a product with a user interface.

2. 02

   Count the output tokens, then ask for fewer

   Layer 08 · Prompt Engineering

   Generation time grows with tokens produced. A tighter output format is often the largest single win and costs nothing to try.

   • Instructor

     Use when

     Schema-validated extraction with typed retry and partial streaming.

     Skip when

     You need a hard token-level guarantee rather than validation.

3. 03

   Find sequential calls that could run in parallel

   Layer 05 · Agent Frameworks

   An agent that calls tools one at a time multiplies its latency by the number of steps. Independent steps should not wait for each other.

   • Pydantic AI

     Use when

     Type-safe agents where schema correctness is the priority.

     Skip when

     You want a batteries-included ecosystem.

   • OpenAI Agents SDK

     Use when

     A small, legible set of primitives for handoffs and guardrails.

     Skip when

     You need multi-provider abstraction at every layer.

4. 04

   Cache repeated prompts and add a faster fallback

   Layer 02 · Routing & Gateways

   A repeated prompt should never reach a model, and a provider having a slow hour should fail over rather than stall every request.

   • LiteLLM

     Use when

     The widest provider coverage, deployed the least committal way.

     Skip when

     You want routing policy to live in a SaaS console.

   • Cloudflare AI Gateway

     Use when

     Edge caching and latency dominate, on Cloudflare already.

     Skip when

     Requests must not transit a third party's network.

   • Portkey

     Use when

     A managed gateway with guardrails and analytics already bundled.

     Skip when

     Prompts cannot leave your infrastructure.

5. 05

   Only then, tune the serving runtime

   Layer 01 · Inference & Serving

   If you self-host and traces show queueing or slow prefill, batching and prefix caching are the levers. Before that, this is the most expensive place to look.

   • vLLM

     Use when

     General GPU serving under bursty, high-concurrency traffic.

     Skip when

     Traffic is steady and low-concurrency, so batches never fill.

   • SGLang

     Use when

     A large share of requests share a long prefix, or you need constrained decoding.

     Skip when

     You want the fewest moving parts and vLLM already clears your bar.

Looks like a fix, is not

• ×Buying a bigger GPU before measuring where the time goes.
• ×Switching provider on a latency benchmark whose prompt lengths look nothing like yours.
• ×Turning off streaming to simplify the client — it makes the wait feel longer.

Quick answers

What should I check first?

Trace one slow request end to end. Latency hides in sequential tool calls, retries and retrieval as often as in generation. You cannot shorten a span you have not seen.

What looks like a fix but is not?

Buying a bigger GPU before measuring where the time goes. Switching provider on a latency benchmark whose prompt lengths look nothing like yours. Turning off streaming to simplify the client — it makes the wait feel longer.