Tensor Cores
Specialized hardware units in NVIDIA GPUs that accelerate matrix multiply-and-accumulate operations used in deep learning.
Tensor Cores are specialized processing units found in NVIDIA GPUs starting with the Volta architecture. They accelerate matrix multiply-and-accumulate operations, which appear throughout neural-network execution. Supported instruction shapes and numerical formats depend on the GPU generation.
GPU kernels combine matrix instructions with other work, such as moving data, applying elementwise operations, and managing intermediate state. Tensor Core throughput contributes to kernel performance alongside memory access, occupancy, and the amount of useful work in each tile.
Tensor Cores support numerical formats such as FP16, BF16, TF32, INT8, and FP8 on compatible GPU generations. Frameworks and inference engines select kernels based on the operations, shapes, and formats they support. Reduced precision should be evaluated for both performance and response quality.
In a dedicated Ion deployment, GPU-kernel tuning is connected to the selected model architecture, precision, and hardware. End-to-end evaluation shows how those kernel choices affect the application's throughput and latency.
Related Terms
Model Quantization
Reducing the numerical precision of a model's weights and activations — from 32-bit to 16, 8, or 4 bits — to shrink memory footprint and speed up memory-bandwidth-bound inference.
Ion
Cumulus' proprietary inference engine, combining model-specific execution and custom GPU kernels in dedicated deployments tuned around the model, hardware, and workload.
KV Cache
A buffer that stores previously computed key and value tensors during autoregressive generation, so each new token only requires computing one step of attention instead of replaying the whole sequence.
Inference
Running a trained model on new inputs to produce outputs — predictions, classifications, generations, embeddings. The production half of machine learning.