CUDA
NVIDIA's parallel computing platform and programming model. The runtime, libraries, and API that let software use NVIDIA GPUs for general-purpose computation.
CUDA (Compute Unified Device Architecture) is the parallel computing platform created by NVIDIA. It includes a runtime, a driver, a set of libraries (cuBLAS, cuDNN, NCCL, TensorRT, and many others), and language extensions for C, C++, and Python. CUDA is what lets PyTorch, TensorFlow, JAX, and every major inference engine run on NVIDIA hardware.
A CUDA program organizes work into kernels — functions that execute in parallel across thousands of GPU threads. Threads are grouped into blocks, blocks into grids, and the CUDA runtime maps blocks onto the GPU's streaming multiprocessors. Developers write logic at the kernel level; the hardware handles fine-grained scheduling.
Most AI engineers never write CUDA directly. They call operations such as `torch.matmul` on GPU tensors, and the framework dispatches GPU kernels. Understanding CUDA helps when debugging performance: a profiler can show time spent on memory access, computation, transfers, and synchronization for a particular workload and device.
CUDA's dominance creates a strong lock-in to NVIDIA hardware. AMD's ROCm and Intel's oneAPI exist as alternatives, but CUDA's mature tooling and library ecosystem keep most production inference on NVIDIA. Inference platforms like Cumulus abstract CUDA management away — driver versions, library compatibility, and kernel selection happen as part of the platform infrastructure.
Related Terms
Tensor Cores
Specialized hardware units in NVIDIA GPUs that accelerate matrix multiply-and-accumulate operations used in deep learning.
Ion
Cumulus' proprietary inference engine, combining model-specific execution and custom GPU kernels in dedicated deployments tuned around the model, hardware, and workload.
vLLM
An open-source LLM serving engine that introduced PagedAttention. The de facto baseline for high-throughput open-weight model serving.
Inference
Running a trained model on new inputs to produce outputs — predictions, classifications, generations, embeddings. The production half of machine learning.