Inference Latency
The end-to-end time from request arrival to response delivery. For LLMs, decomposed into time-to-first-token (TTFT) and inter-token latency (ITL).
Inference latency measures the total time required to process a single inference request, from the moment it arrives at the serving endpoint to the moment the response is returned. For user-facing applications it directly determines how responsive the system feels. It is typically tracked at multiple percentiles — P50 for the typical experience, P95 and P99 for the tail — because a small fraction of slow requests can dominate user perception.
For autoregressive language models, **time-to-first-token (TTFT)** measures the interval from request start to the first output token, including queueing and prompt processing. **Inter-token latency (ITL)** measures the time between subsequent generated tokens. End-to-end latency includes the wait for the first token and the time spent generating the remaining output.
Latency depends on prompt length, output length, model architecture, hardware, and traffic. Scheduling and batching affect both capacity and the time individual requests spend waiting or executing. Measure latency across the expected load range, including the slowest requests at relevant percentiles.
Model execution, GPU kernels, caching, and scheduling are all useful places to investigate latency. In a dedicated Ion deployment, those choices are evaluated around the model and workload. Compare completed-request throughput within the application's latency targets, and measure model-loading or switching time separately from warm serving.
Related Terms
KV Cache
A buffer that stores previously computed key and value tensors during autoregressive generation, so each new token only requires computing one step of attention instead of replaying the whole sequence.
vLLM
An open-source LLM serving engine that introduced PagedAttention. The de facto baseline for high-throughput open-weight model serving.
Prompt Cache
A response cache keyed on the request — exact-match for identical requests, prefix for shared system prompts, and optionally semantic for paraphrased questions. Cuts input tokens dramatically on real traffic.
Ion
Cumulus' proprietary inference engine, combining model-specific execution and custom GPU kernels in dedicated deployments tuned around the model, hardware, and workload.
Inference
Running a trained model on new inputs to produce outputs — predictions, classifications, generations, embeddings. The production half of machine learning.