vLLM
An open-source LLM serving engine that introduced PagedAttention. The de facto baseline for high-throughput open-weight model serving.
vLLM is an open-source library for fast and efficient large language model inference and serving. Originally developed at UC Berkeley, it has become one of the most widely adopted LLM serving engines because of its strong throughput, broad model coverage, and OpenAI-compatible API. vLLM supports LLaMA, Mistral, Qwen, GPT-NeoX, Falcon, and most other major architectures.
The key innovation in vLLM is **PagedAttention**, an attention algorithm inspired by virtual memory paging in operating systems. Reserving a large contiguous KV-cache allocation for each request can waste GPU memory. PagedAttention stores the cache in dynamically allocated, non-contiguous blocks, reducing that waste and making room for more concurrent requests.
Beyond PagedAttention, vLLM ships **continuous batching**, which adds new requests as active requests finish. A short request can release its slot without waiting for the longest request in a fixed batch, helping the engine use available capacity as response lengths vary.
vLLM is a useful baseline when evaluating serving engines such as SGLang and Cumulus' Ion. Compare the same model, precision, hardware, and traffic pattern, and measure throughput alongside response latency and quality. The useful result is the configuration that meets the application's requirements.
Related Terms
SGLang
An open-source LLM serving engine that uses RadixAttention to share KV cache across requests with overlapping prefixes — especially strong for agentic and tool-use workloads.
KV Cache
A buffer that stores previously computed key and value tensors during autoregressive generation, so each new token only requires computing one step of attention instead of replaying the whole sequence.
Batch Inference
Processing multiple inference requests in the same forward pass to maximize GPU throughput and hardware utilization.
Ion
Cumulus' proprietary inference engine, combining model-specific execution and custom GPU kernels in dedicated deployments tuned around the model, hardware, and workload.
Inference
Running a trained model on new inputs to produce outputs — predictions, classifications, generations, embeddings. The production half of machine learning.