Lattice
Skip to content
← Comparisons

vLLM vs llama.cpp

This pairing gets compared constantly, and the comparison is usually a category error. vLLM is a GPU server built for concurrent requests. llama.cpp is a CPU-first runtime built to run on whatever hardware you already have. The question is not which is faster in the abstract. It is which job you have.

Short answer

Use vLLM when you are serving a model to more than a handful of concurrent users on a GPU you control: its paged attention and continuous batching exist for exactly that load.

By Kunj Shah · Licence and cost facts verified · method

vLLM vs llama.cpp compared across 6 dimensions
DimensionvLLMllama.cpp
Best forBursty, high-concurrency GPU servingNo GPU, an edge box, or a laptop
Hardware floorOne CUDA GPU, minimumNone. CPU alone is the default path
Model formatSafetensors, at full or reduced precisionGGUF, quantised down to 2-bit
ConcurrencyContinuous batching, built for many requests at onceServes one request well; many at once is not the design centre
APIOpenAI-compatible server built inOpenAI-compatible server (llama-server) built in
Where it losesNo GPU, or no budget for oneMaximum concurrent throughput on datacentre GPUs

Scroll the table sideways to see every tool.

What this table is not. These are editorial judgements, not benchmarks: Lattice has not run these tools head to head, and no cell uses GitHub stars or vendor benchmark claims as evidence. Licence and cost facts are re-checked on a schedule the build enforces — the receipt is public.

The recommendation

Which should you choose: vLLM or llama.cpp?

Use vLLM when you are serving a model to more than a handful of concurrent users on a GPU you control: its paged attention and continuous batching exist for exactly that load. Use llama.cpp when the constraint is hardware rather than throughput, such as no GPU, an edge device, or a laptop you want a model running on today. The two rarely compete for the same deployment, and the decision usually resolves itself the moment you say out loud whether a GPU is in the budget.

Rules of thumb

  • If a GPU is in the budget and the traffic is bursty, vLLM wins the decision before you benchmark anything.
  • GGUF quantisation is why llama.cpp runs on a laptop at all. The same model at full precision would not fit.
  • Running llama.cpp on a GPU is possible, and sometimes right for a single user, but it is not the throughput path. That is what vLLM is for.

Quick answers

When should you use vLLM, and when should you skip it?
Use vLLM when: general GPU serving under bursty, high-concurrency traffic. Skip it when: traffic is steady and low-concurrency, so batches never fill.
When should you use llama.cpp, and when should you skip it?
Use llama.cpp when: the constraint is hardware, not throughput: no GPU, or an edge box. Skip it when: you need maximum concurrent throughput on datacentre GPUs.

Ask an AI about this page

Opens the assistant with this page’s link and a question already filled in.