vLLM
vllm.airuntimeself-hosted
vLLM is a self-hosted runtime in the Inference & Serving layer of the AI stack. Paged-attention inference engine with an OpenAI-compatible server.
By Lattice editorial · Facts verified · method
- licence
- Apache-2.0
- language
- Python
- cost
- free
- owned by
- AI Infrastructure
Verified 2026-09build refuses it from 2027-04
Use vLLM when
General GPU serving under bursty, high-concurrency traffic.
Skip vLLM when
Traffic is steady and low-concurrency, so batches never fill.
Turns weights into throughput. — that is what this layer of the stack is answerable for. See the full Inference & Serving section for the rest of the options.
Visit vllm.aiCompare with another toolBadge this entry
[](https://lattice.kunjshah.in/inference-serving/vllm)Quick answers
- Is vLLM open source?
- Yes. vLLM is open source under the Apache-2.0 licence. It is free to use.
- What are the alternatives to vLLM?
- The closest alternatives to vLLM are SGLang, Text Generation Inference and Triton Inference Server.
Compared in
- vLLM vs SGLang vs TGI vs llama.cppThe four open inference runtimes compared on prefix reuse, hardware floor, structured output and operational complexity — with a recommendation for each traffic shape.
- Self-hosting vs routing vs measuring: cutting inference costThree ways to lower an inference bill — run the model yourself, send easy queries to a smaller model, or measure where spend goes — compared on savings and risk.
Reading on inference
- Choosing an inference runtimeThe wrong runtime costs you throughput you never knew you were leaving on the table. The right one is usually decided by traffic shape, not model size.
- Context is a budget, not a containerThe instinct is to keep everything in case it is needed. The models are not filing systems, and past a certain length the middle of your context is doing nothing but costing money.
- Choosing a model is a routing problemThere is no best model. There is a best model for a task, at a quality bar, within a latency and cost budget — and that decision is a routing policy, not a benchmark.
Alternatives to vLLM
compare 3 →Listed as an alternative to
- SGLangSGLangStructured generation runtime built on RadixAttention prefix reuse.Use when A large share of requests share a long prefix, or you need constrained decoding.Skip when You want the fewest moving parts and vLLM already clears your bar.Apache-2.0 · self-hosted · free
- TensorRT-LLMTensorRT-LLMNVIDIA-optimized inference with inflight batching and FP8.Use when You are committed to NVIDIA hardware and need the last of the throughput.Skip when Hardware portability matters, or you have no GPUs to tune against.Apache-2.0 · self-hosted · free
- Text Generation InferenceText Generation InferenceHugging Face production server for LLMs with tensor parallelism.Use when Multi-GPU tensor parallelism inside an existing Hugging Face stack.Skip when The model fits on one GPU, where this is overkill.Apache-2.0 · self-hosted · free
- Triton Inference ServerTriton Inference ServerMulti-framework inference server for ONNX, TensorRT and Python.Use when One endpoint serving ONNX, TensorRT and Python models side by side.Skip when You serve one model family and want peak LLM throughput.BSD-3-Clause · self-hosted · free
- PowerInferPowerInferCPU-first runtime that offloads hot paths to the GPU.Use when CPU-first serving where the GPU accelerates rather than carries the load.Skip when You have GPUs to spare — it trades peak throughput for reach.Apache-2.0 · self-hosted · free
Also in inference
- SGLangSGLangStructured generation runtime built on RadixAttention prefix reuse.Use when A large share of requests share a long prefix, or you need constrained decoding.Skip when You want the fewest moving parts and vLLM already clears your bar.Apache-2.0 · self-hosted · free
- llama.cppllama.cppPortable CPU/GPU inference for quantized GGUF models.Use when The constraint is hardware, not throughput — no GPU, or an edge box.Skip when You need maximum concurrent throughput on datacentre GPUs.MIT · self-hosted · free
- OllamaOllamaLocal model runner with a single-binary distribution and HTTP API.Use when Local model access for a developer or small team, with no setup.Skip when You need production throughput or a stable server API.MIT · self-hosted · free
- TensorRT-LLMTensorRT-LLMNVIDIA-optimized inference with inflight batching and FP8.Use when You are committed to NVIDIA hardware and need the last of the throughput.Skip when Hardware portability matters, or you have no GPUs to tune against.Apache-2.0 · self-hosted · free
- Text Generation InferenceText Generation InferenceHugging Face production server for LLMs with tensor parallelism.Use when Multi-GPU tensor parallelism inside an existing Hugging Face stack.Skip when The model fits on one GPU, where this is overkill.Apache-2.0 · self-hosted · free
- LM StudioLM StudioDesktop app for running and serving local models with an OpenAI API.Use when A desktop GUI for running and serving local models, aimed at one operator.Skip when You need headless, multi-tenant deployment.proprietary · self-hosted · free
- Triton Inference ServerTriton Inference ServerMulti-framework inference server for ONNX, TensorRT and Python.Use when One endpoint serving ONNX, TensorRT and Python models side by side.Skip when You serve one model family and want peak LLM throughput.BSD-3-Clause · self-hosted · free
- llamafilellamafilePackages a model and its runtime into a single executable.Use when Shipping a model as a single executable with no install step.Skip when You need batching, multi-GPU serving or a production HTTP server.Apache-2.0 · self-hosted · free
- KoboldCppKoboldCppGGUF inference with a focused text-adventure UI and API.Use when Local generation with a UI tuned for interactive, long-form use.Skip when You need headless serving at scale.MIT · self-hosted · free
- PowerInferPowerInferCPU-first runtime that offloads hot paths to the GPU.Use when CPU-first serving where the GPU accelerates rather than carries the load.Skip when You have GPUs to spare — it trades peak throughput for reach.Apache-2.0 · self-hosted · free
- MarlinMarlinQuantised GEMM kernels for near-GPU speed at 4-bit.Use when Near-GPU throughput from 4-bit weights, as a kernel inside another runtime.Skip when You need a scheduler and a server as well as a kernel.Apache-2.0 · self-hosted · free