ONNX Runtime Mobile

Executes ONNX-format neural network models directly on ARM processors in iOS and Android devices using native CPU execution providers with operator-level optimization for mobile instruction sets. The runtime compiles ONNX graph operations into ARM-native code paths, avoiding cloud round-trips and enabling sub-100ms latency inference on commodity mobile hardware.

Routes compatible ONNX operations to platform-native acceleration frameworks—CoreML on iOS, NNAPI on Android, and XNNPACK for CPU-based SIMD optimization on both platforms—while automatically falling back to CPU execution for unsupported operators. The runtime partitions the computation graph, sending accelerator-compatible subgraphs to specialized hardware and executing remaining operations on the CPU.

Provides APIs to measure inference latency, memory usage, and operator-level execution time. Developers can enable profiling at session creation time to collect per-operator timing and memory allocation data. Profiling output includes execution provider information (which provider executed each operator) and can be used to identify performance bottlenecks.

Implements memory optimization techniques including operator fusion (combining multiple operators into single kernel), memory planning (pre-allocating buffers for intermediate activations), and memory reuse (reusing buffers across operators). Developers can configure memory optimization level through SessionOptions to trade off memory usage vs. optimization overhead.

Validates ONNX model format, operator compatibility, and tensor shapes at session creation and inference time. The runtime returns error codes and messages for invalid models, unsupported operators, and shape mismatches. Error handling is language-specific (exceptions in Java/C#, error codes in C++).

Supports loading and executing quantized ONNX models (8-bit integer weights and activations) that reduce model size by ~4x compared to 32-bit float models, enabling larger models to fit in device memory and storage constraints. The runtime executes quantized operations natively on ARM processors and delegates to accelerators (NNAPI, CoreML) which have native quantized operation support.

Provides language-specific SDKs for iOS (C/C++, Objective-C), Android (Java, C, C++), and cross-platform (C# via MAUI/Xamarin) that wrap the core ONNX Runtime inference engine with idiomatic APIs for each platform. Each SDK exposes session management, input/output tensor handling, and execution provider configuration through language-native abstractions.

Exposes SessionOptions API allowing developers to configure inference behavior including execution provider priority (CPU, CoreML, NNAPI, XNNPACK), thread pool size, memory optimization flags, and operator-level profiling. The runtime uses a priority-ordered list of execution providers, attempting to use the first available provider and falling back to the next if operators are unsupported.

Allows developers to register custom C++ operators that extend ONNX Runtime's built-in operator library, enabling inference of models with domain-specific or experimental operations. Custom operators are registered at session creation time and executed through the same inference pipeline as built-in operators, with support for custom execution providers.

Loads ONNX models from multiple sources: file system paths, in-memory byte arrays, and memory-mapped files. The runtime validates model format, parses the ONNX graph, and initializes the inference session with minimal overhead. Supports both synchronous loading and asynchronous loading patterns for non-blocking model initialization.

Manages tensor creation, shape validation, and data marshalling between application code and the inference engine. The runtime validates input tensor shapes against model expectations, allocates output tensors, and handles data type conversions (float32, int32, int64, uint8). Supports both pre-allocated tensors and automatic tensor allocation.

Executes inference on input tensors, returning output tensors with results. The runtime supports single-instance inference (batch size 1) and explicit batching (batch size > 1) where multiple inputs are processed in a single forward pass. Execution is synchronous; asynchronous execution is not supported.

Supports inference of models converted from PyTorch, TensorFlow, TFLite, and scikit-learn to ONNX format. The runtime does not perform conversion itself; conversion is done externally using tools like ONNX exporter, TensorFlow-ONNX, or skl2onnx. Once converted to ONNX, models are loaded and executed through the standard inference pipeline.