AI Infrastructure

Use when General GPU serving under bursty, high-concurrency traffic.

Skip when Traffic is steady and low-concurrency, so batches never fill.

Use when A large share of requests share a long prefix, or you need constrained decoding.

Skip when You want the fewest moving parts and vLLM already clears your bar.

Use when The constraint is hardware, not throughput — no GPU, or an edge box.

Skip when You need maximum concurrent throughput on datacentre GPUs.

Use when Local model access for a developer or small team, with no setup.

Skip when You need production throughput or a stable server API.

Use when You are committed to NVIDIA hardware and need the last of the throughput.

Skip when Hardware portability matters, or you have no GPUs to tune against.

Use when Multi-GPU tensor parallelism inside an existing Hugging Face stack.

Skip when The model fits on one GPU, where this is overkill.

Use when A desktop GUI for running and serving local models, aimed at one operator.

Skip when You need headless, multi-tenant deployment.

Use when One endpoint serving ONNX, TensorRT and Python models side by side.

Skip when You serve one model family and want peak LLM throughput.

Use when Shipping a model as a single executable with no install step.

Skip when You need batching, multi-GPU serving or a production HTTP server.

Use when Local generation with a UI tuned for interactive, long-form use.

Skip when You need headless serving at scale.

Use when CPU-first serving where the GPU accelerates rather than carries the load.

Skip when You have GPUs to spare — it trades peak throughput for reach.

Use when Near-GPU throughput from 4-bit weights, as a kernel inside another runtime.

Skip when You need a scheduler and a server as well as a kernel.

Use when A Kubernetes shop that already runs Envoy and needs the gateway in-cluster.

Skip when You want a gateway configurable without a data plane.

Use when LoRA fine-tuning on a single GPU, with memory and time roughly halved.

Skip when You need a training stack you can audit line by line.

Use when ZeRO sharding and pipeline parallelism across many GPUs.

Skip when The model fits on one device — the complexity buys nothing.

Use when Tensor and pipeline parallel pretraining at cluster scale.

Skip when You are fine-tuning rather than pretraining.

Use when Hosting stateful containers that fits durable agent workers.

Skip when You want to own the infrastructure layer.

Use when Serverless GPUs and containers for batch inference and scheduled jobs.

Skip when Data locality rules forbid ephemeral compute.