Model Quantization
Reducing the numerical precision of a model's weights and activations — from 32-bit to 16, 8, or 4 bits — to shrink memory footprint and speed up memory-bandwidth-bound inference.
Quantization represents model weights or activations at reduced precision, using formats such as INT8 or INT4. At four bits per weight, 70 billion weights occupy about 35 GB before scales, metadata, and other runtime storage. Weight storage is one part of the total memory required to serve the model.
Quantization reduces the bytes needed for weights or activations and can make a model fit on fewer or smaller GPUs. Performance depends on the available kernels, hardware, batch size, and cost of converting or unpacking values. Evaluate memory use, latency, throughput, and task quality together for the chosen configuration.
Methods such as **GPTQ** and **AWQ** use calibration data to guide weight quantization. Quantization-aware training (QAT) accounts for reduced precision during training. The resulting quality depends on the model, method, calibration or training data, and task being evaluated.
Serving engines support particular combinations of model architecture, quantization format, and hardware. Ion's dedicated-deployment work includes choosing numerical precision alongside execution paths and GPU kernels. That choice should be evaluated on representative inputs and the application's quality requirements.
Related Terms
KV Cache
A buffer that stores previously computed key and value tensors during autoregressive generation, so each new token only requires computing one step of attention instead of replaying the whole sequence.
Model Weights
The learned numerical parameters of a neural network, stored as large multi-dimensional arrays. The artifact that defines what a trained model does.
vLLM
An open-source LLM serving engine that introduced PagedAttention. The de facto baseline for high-throughput open-weight model serving.
Ion
Cumulus' proprietary inference engine, combining model-specific execution and custom GPU kernels in dedicated deployments tuned around the model, hardware, and workload.
Inference
Running a trained model on new inputs to produce outputs — predictions, classifications, generations, embeddings. The production half of machine learning.