Batch Inference
Processing multiple inference requests in the same forward pass to maximize GPU throughput and hardware utilization.
Batch inference groups multiple inputs and processes them together in a single forward pass through a model. Instead of running one input at a time, the GPU processes a batch of inputs simultaneously, exploiting its parallel architecture to achieve much higher throughput than sequential processing.
The efficiency gains come from GPU architecture. A single small input may leave much of the device idle. Batching increases the amount of parallel work and can amortize weight access and kernel-launch overhead across inputs. The resulting throughput depends on the model, input shapes, batch size, and hardware.
There are two common batching approaches for serving. **Static batching** forms a fixed group of requests to process together. **Continuous batching** admits new requests as active requests finish, so available slots can be reused without waiting for every member of an initial group. Serving engines combine batching with scheduling and memory management.
Batch size is part of a latency-throughput tradeoff. Queueing to form a batch and execution within that batch both affect response time. Memory also constrains capacity through weights, activations, workspaces, and request state such as KV caches. Production deployments tune batching against their workload and response-time targets.
Related Terms
vLLM
An open-source LLM serving engine that introduced PagedAttention. The de facto baseline for high-throughput open-weight model serving.
SGLang
An open-source LLM serving engine that uses RadixAttention to share KV cache across requests with overlapping prefixes — especially strong for agentic and tool-use workloads.
KV Cache
A buffer that stores previously computed key and value tensors during autoregressive generation, so each new token only requires computing one step of attention instead of replaying the whole sequence.
Inference
Running a trained model on new inputs to produce outputs — predictions, classifications, generations, embeddings. The production half of machine learning.