Inference & Serving

Turns weights into throughput.

• vLLMruntimePaged-attention inference engine with an OpenAI-compatible server.vllm.aiMore about vLLM in this index
• SGLangruntimeStructured generation runtime built on RadixAttention prefix reuse.github.comMore about SGLang in this index
• llama.cppruntimePortable CPU/GPU inference for quantized GGUF models.github.comMore about llama.cpp in this index
• OllamaruntimeLocal model runner with a single-binary distribution and HTTP API.ollama.comMore about Ollama in this index
• TensorRT-LLMruntimeNVIDIA-optimized inference with inflight batching and FP8.github.comMore about TensorRT-LLM in this index
• Text Generation InferenceruntimeHugging Face production server for LLMs with tensor parallelism.github.comMore about Text Generation Inference in this index
• LM StudioserviceDesktop app for running and serving local models with an OpenAI API.lmstudio.aiMore about LM Studio in this index
• Triton Inference ServerruntimeMulti-framework inference server for ONNX, TensorRT and Python.github.comMore about Triton Inference Server in this index
• llamafileruntimePackages a model and its runtime into a single executable.github.comMore about llamafile in this index
• KoboldCppruntimeGGUF inference with a focused text-adventure UI and API.github.comMore about KoboldCpp in this index
• PowerInferruntimeCPU-first runtime that offloads hot paths to the GPU.github.comMore about PowerInfer in this index
• MarlinlibraryQuantised GEMM kernels for near-GPU speed at 4-bit.github.comMore about Marlin in this index