SGLang
An open-source LLM serving engine that uses RadixAttention to share KV cache across requests with overlapping prefixes — especially strong for agentic and tool-use workloads.
SGLang is an open-source structured generation language and serving engine for large language models. Its **RadixAttention** organizes KV-cache reuse across requests that share prefixes. Paged allocation and prefix reuse address different parts of cache management and can be used together.
The core observation behind RadixAttention is that many workloads send requests with substantial overlap — the same system prompt, few-shot examples, or tool definitions. SGLang organizes cached prefixes in a radix tree and matches incoming requests against them. Reusing a matching prefix's KV state reduces prompt processing; the model still generates the new response.
Prefix reuse is useful in agentic workflows with shared tool prompts, evaluations with repeated instructions, and applications that reuse the same context. Its benefit depends on how often prefixes actually match and how much of that state remains cached.
When comparing SGLang, vLLM, and a custom engine such as Ion, include the application's real prompt-sharing pattern and cache policy. Measure completed-request throughput, latency, and response quality under the same conditions.
Related Terms
vLLM
An open-source LLM serving engine that introduced PagedAttention. The de facto baseline for high-throughput open-weight model serving.
KV Cache
A buffer that stores previously computed key and value tensors during autoregressive generation, so each new token only requires computing one step of attention instead of replaying the whole sequence.
Prompt Cache
A response cache keyed on the request — exact-match for identical requests, prefix for shared system prompts, and optionally semantic for paraphrased questions. Cuts input tokens dramatically on real traffic.
Ion
Cumulus' proprietary inference engine, combining model-specific execution and custom GPU kernels in dedicated deployments tuned around the model, hardware, and workload.
Inference
Running a trained model on new inputs to produce outputs — predictions, classifications, generations, embeddings. The production half of machine learning.