Definition
ONNX (Open Neural Network Exchange) is an open intermediate model format that supports interoperability of models exported from mainstream frameworks such as TensorFlow, PyTorch, and Caffe. After being converted via ONNX, models can be deployed to edge devices, NPU, or GPU, and optimized at runtime with TensorRT and similar tools to achieve cross-platform inference. It is suitable for edge AI scenarios such as industrial vision, defect detection, and OCR.
Workflow
The training stage is completed in frameworks like PyTorch or TensorFlow; then the model is exported in ONNX format (.onnx file); ONNX Runtime, TensorRT, OpenVINO, and various NPU SDKs can load ONNX for inference. ONNX supports operator extensions, and some custom operators need to be adapted for each runtime. Model optimizations (quantization, pruning, operator fusion) can be performed at either the ONNX or runtime level, and TensorRT can further optimize the ONNX model into an engine file.
Use Cases
Industrial visual defect detection, object detection, classification; security facial recognition, behavior analysis; OCR text recognition; speech recognition. Edge AI gateways, industrial computers, and embedded devices can load ONNX models through ONNX Runtime or vendor SDKs to perform local inference, reducing latency and bandwidth requirements.
Intermediate Representation and Runtime Differences
The graph stores operators, not training
An open exchange format stores a computation graph that has already been fixed. Nodes are operators, edges are tensor shapes and data types, and weights are written in the file as initializers. Optimizer state, data augmentation, and the loss function do not travel with it. What it solves is whether another inference program can run the same graph. It does not move the whole training project. The same file can be read by different runtimes, but each runtime guarantees only the operators it has implemented. Opening the file does not mean every layer executes on the target chip.
The graph also records an opset version. The version decides how an operator's attributes are interpreted: padding, scale coordinates, or whether a dynamic axis is allowed. If the exporter writes a new version and the runtime knows only an old one, loading reports a missing operator, or worse, silently computes a different number under the old semantics. The deployment package should keep the export log and the opset number together. Copying a file with the right extension is not enough.
Fix the input shape before export
The premise for using this intermediate file is that the input tensor at inference can be stated at export time. If image height and width, batch, and sequence length are left fully dynamic, the downstream compiler often refuses the graph, or can take only a very slow fallback path. Before export, run inference once at the real resolution used on site, and record the preprocessing color order, normalization coefficients, and layout in the same note. Changing layout from channels-first to channels-last shifts every value, and the file itself reports no error.
It suits a model whose structure is already frozen and that needs to be handed between several inference backends. If the network is still changing often, or it depends on a custom CUDA operator inside the training framework, export drops layers. Replace custom operators with an equivalent that can be exported before talking about putting the model on a board. If the receiving end is a compiler that consumes only a private engine, the intermediate file is only the entrance. What actually runs is a different artifact after compilation. Both need version control.
Version alignment and custom operators
A common error is to treat a successful desktop inference run as proof the board matches. The desktop runtime falls back to the framework for an operator it has not implemented. The board has no such framework, and loading fails immediately. Compare layer by layer on the target runtime. At least run a few site images and look at the output difference. Do not stop at file size. Another error is preprocessing that lives in the training code but was never written into the export note. The board uses a different scale, accuracy drops, and the drop is misread as a quantization problem.
Weights are folded into constants during conversion. Changing one or two thresholds afterward cannot be patched in place. The model has to be exported again. Several engineers exporting separately, with the same file name and different opsets, make device behavior follow "whichever copy arrived last." Archive the export command, the input sample, and the runtime version together, or the same inference result cannot be reproduced months later.
Quantization parameters should not be recomputed ad hoc on the board either. Once the calibration set changes, the scales change with it, the output distribution of the same site image shifts as a whole, and the old thresholds all fail. Archive the calibration-set version with the thresholds. When the camera or the lighting changes, redo calibration on purpose, rather than waiting for false alarms to rise and then guessing that the model is broken.