ONNX Runtime

Executes ONNX models across heterogeneous hardware (CPU, CUDA GPUs, TensorRT, DirectML, CoreML, OpenVINO, NPU) through a pluggable execution provider architecture. Each provider implements a standardized interface that abstracts hardware-specific optimizations, with automatic fallback to CPU kernels when specialized hardware is unavailable. The provider bridge pattern routes operations to the optimal hardware target based on session configuration and operator support.

Analyzes the ONNX computation graph to identify optimization opportunities including operator fusion (combining multiple ops into single fused kernels), constant folding (pre-computing operations on static inputs), and dead code elimination. The optimizer traverses the graph using a visitor pattern, applies provider-specific optimization passes, and reconstructs an optimized graph that reduces memory bandwidth and kernel launch overhead. Optimizations are applied during session initialization before inference begins.

Collects per-operator execution time, memory allocation, and kernel launch overhead during inference. Profiling is enabled via session options and generates detailed timeline data showing which operators consume the most time/memory. Profiler output can be exported to JSON or Chrome tracing format for visualization. Supports both wall-clock time and GPU-specific metrics (CUDA kernel time, memory transfers). Profiling adds ~5-10% overhead; intended for development/optimization, not production.

Saves and loads ONNX models in standard .onnx format (protobuf-based). Supports saving optimized graphs (after graph optimization) for faster subsequent loading. Enables checkpoint management for training workflows: saving model weights and optimizer state, loading checkpoints to resume training. Serialization preserves all model metadata (operator schemas, initializers, attributes) enabling round-trip compatibility.

Supports ONNX models with dynamic (variable) input shapes by performing symbolic shape inference at load time and runtime shape validation during inference. Dynamic shapes are represented as symbolic dimensions (e.g., 'batch_size' instead of fixed integer). Graph optimization is conservative for dynamic shapes to avoid invalid assumptions. At inference time, actual input shapes are validated against model constraints and used to allocate output tensors. Supports partial dynamic shapes (some dimensions fixed, others dynamic).

Executes quantized ONNX models (INT8, UINT8, FLOAT16) with specialized quantized kernels that perform computation in lower precision while maintaining accuracy through learned quantization parameters (scale, zero-point). Supports mixed-precision graphs where some operations run in FP32 and others in INT8, with automatic type conversion at boundaries. Quantized operators are registered separately from standard operators and optimized for target hardware (e.g., VNNI instructions on CPU, Tensor Cores on NVIDIA GPUs).

Allows developers to register custom ONNX operators (not in standard opset) by implementing a kernel interface and registering it with the operator registry. Custom operators are compiled into shared libraries (.so/.dll) and loaded at runtime, then executed through the same inference pipeline as built-in operators. Supports both CPU and GPU custom kernels with provider-specific implementations. The operator registration system uses a factory pattern to instantiate kernels based on operator type and execution provider.

Manages tensor memory allocation and deallocation through a pluggable allocator interface, supporting both CPU memory (malloc-based) and GPU memory (CUDA, DirectML). IOBinding enables zero-copy inference by allowing users to pre-allocate input/output tensors and bind them directly to the inference session, eliminating intermediate allocations. Memory is managed per-session with configurable arena allocators that pre-allocate large blocks to reduce fragmentation. Supports memory mapping for large models to reduce peak memory usage.

Provides optimized CPU kernels for common operations (GEMM, element-wise ops, quantized ops) using SIMD instructions (AVX2, AVX-512, NEON on ARM). MLAS (Microsoft Linear Algebra Subroutines) is a thin abstraction over platform-specific SIMD code, with runtime CPU feature detection to select optimal kernel variants. Implements specialized kernels for quantized operations (GEMM with INT8 inputs), attention mechanisms, and other performance-critical operations. Kernels are hand-optimized assembly or intrinsics for maximum performance.

Loads ONNX model files (.onnx format) and parses the protobuf graph structure into an in-memory graph representation. Performs shape inference to compute output tensor shapes based on input shapes and operator semantics, enabling memory pre-allocation and optimization. Validates model against ONNX specification (opset version, operator schemas, type compatibility). Supports model loading from file, memory buffer, or custom I/O interface. Graph is represented as a DAG (directed acyclic graph) with nodes (operators) and edges (tensors).

Creates an InferenceSession object that encapsulates a loaded ONNX model and execution configuration. Session initialization includes graph optimization, memory allocation, and execution provider initialization. Supports session options to control behavior: execution provider priority order, graph optimization level, memory arena settings, inter/intra-op threading, and profiling. Provider selection is automatic based on availability and priority order; unavailable providers are skipped with fallback to next in priority list. Session is thread-safe for concurrent inference calls.

Integrates ONNX Runtime into PyTorch training pipelines via ORTModule, which wraps a PyTorch model and executes the forward pass using ONNX Runtime while maintaining PyTorch's autograd for backward pass. Exports PyTorch model to ONNX format, builds a gradient graph for backpropagation, and optimizes both forward and backward graphs. Supports mixed-precision training with automatic loss scaling. Enables training acceleration through ONNX Runtime's graph optimizations and execution providers (CUDA, TensorRT).

Provides language bindings for C/C++, Python, C#, and JavaScript/Node.js with consistent API semantics across languages. C API is the lowest-level interface (onnxruntime_c_api.h) providing ABI stability; C++ API wraps C API with RAII and exceptions; Python bindings use ctypes to call C API; C# uses P/Invoke. All bindings expose the same core functionality: session creation, model loading, inference execution, and profiling. Language-specific idioms are preserved (e.g., NumPy arrays in Python, Tensors in C++).