vLLM vs llama.cpp
This pairing gets compared constantly, and the comparison is usually a category error. vLLM is a GPU server built for concurrent requests. llama.cpp is a CPU-first runtime built to run on whatever hardware you already have. The question is not which is faster in the abstract. It is which job you have.
Short answer
Use vLLM when you are serving a model to more than a handful of concurrent users on a GPU you control: its paged attention and continuous batching exist for exactly that load.
By Kunj Shah · Licence and cost facts verified · method
- vLLMGPU server. Paged attention, built for concurrent requests.
- llama.cppCPU-first. Quantised GGUF models, runs on almost anything.
| Dimension | vLLM | llama.cpp |
|---|---|---|
| Best for | Bursty, high-concurrency GPU serving | No GPU, an edge box, or a laptop |
| Hardware floor | One CUDA GPU, minimum | None. CPU alone is the default path |
| Model format | Safetensors, at full or reduced precision | GGUF, quantised down to 2-bit |
| Concurrency | Continuous batching, built for many requests at once | Serves one request well; many at once is not the design centre |
| API | OpenAI-compatible server built in | OpenAI-compatible server (llama-server) built in |
| Where it loses | No GPU, or no budget for one | Maximum concurrent throughput on datacentre GPUs |
Scroll the table sideways to see every tool.
What this table is not. These are editorial judgements, not benchmarks: Lattice has not run these tools head to head, and no cell uses GitHub stars or vendor benchmark claims as evidence. Licence and cost facts are re-checked on a schedule the build enforces — the receipt is public.
The recommendation
Which should you choose: vLLM or llama.cpp?
Use vLLM when you are serving a model to more than a handful of concurrent users on a GPU you control: its paged attention and continuous batching exist for exactly that load. Use llama.cpp when the constraint is hardware rather than throughput, such as no GPU, an edge device, or a laptop you want a model running on today. The two rarely compete for the same deployment, and the decision usually resolves itself the moment you say out loud whether a GPU is in the budget.
Rules of thumb
- If a GPU is in the budget and the traffic is bursty, vLLM wins the decision before you benchmark anything.
- GGUF quantisation is why llama.cpp runs on a laptop at all. The same model at full precision would not fit.
- Running llama.cpp on a GPU is possible, and sometimes right for a single user, but it is not the throughput path. That is what vLLM is for.
Quick answers
- When should you use vLLM, and when should you skip it?
- Use vLLM when: general GPU serving under bursty, high-concurrency traffic. Skip it when: traffic is steady and low-concurrency, so batches never fill.
- When should you use llama.cpp, and when should you skip it?
- Use llama.cpp when: the constraint is hardware, not throughput: no GPU, or an edge box. Skip it when: you need maximum concurrent throughput on datacentre GPUs.
Ask an AI about this page
Opens the assistant with this page’s link and a question already filled in.