Ion
Cumulus' proprietary inference engine, combining model-specific execution and custom GPU kernels in dedicated deployments tuned around the model, hardware, and workload.
Ion is the proprietary inference engine built by Cumulus Labs. It combines model-specific execution paths, custom GPU kernels, and memory management. Cumulus tunes dedicated deployments around the selected models, numerical precision, hardware, and expected traffic.
IonAttention is part of Ion's work on attention kernels. On Grace Hopper, the broader runtime also coordinates GPU HBM and CPU LPDDR memory through NVLink-C2C. GPU kernel execution, movement of model weights, and KV-cache management each contribute to how the deployment serves requests.
A dedicated deployment can load supported models from system memory onto the same GPU as requests change. Its model set and memory budget are configured together, allowing models to reuse GPU capacity without all remaining resident at once.
IonRouter is the public service for accessing hosted models through an OpenAI-compatible API with usage-based pricing. Dedicated Ion deployments are available on Cumulus infrastructure or supported customer hardware. To evaluate a deployment, start with the model versions, traffic pattern, response quality, and latency targets, then measure throughput within those requirements.
Related Terms
vLLM
An open-source LLM serving engine that introduced PagedAttention. The de facto baseline for high-throughput open-weight model serving.
SGLang
An open-source LLM serving engine that uses RadixAttention to share KV cache across requests with overlapping prefixes — especially strong for agentic and tool-use workloads.
KV Cache
A buffer that stores previously computed key and value tensors during autoregressive generation, so each new token only requires computing one step of attention instead of replaying the whole sequence.
Inference
Running a trained model on new inputs to produce outputs — predictions, classifications, generations, embeddings. The production half of machine learning.