Definition
TensorRT is a deep learning inference optimization engine developed by NVIDIA. Through techniques such as INT8 quantization, layer fusion, and automatic kernel tuning, it can increase inference speed by 2–3 times while significantly reducing computing power requirements without sacrificing accuracy. It supports ONNX model import and is suitable for NVIDIA hardware such as Jetson and data center GPUs. When combined with GPU edge devices, it enables efficient inference for scenarios like industrial vision and defect detection.
Optimization techniques
TensorRT converts models like ONNX, TensorFlow, and PyTorch into optimized engines: layer fusion reduces kernel launch overhead; precision calibration (INT8) speeds up computation while keeping accuracy loss controllable; automatic kernel tuning selects the best implementation; dynamic shape support allows variable input. The output is a .engine or .plan file, which is loaded and executed during inference. Jetson series edge devices commonly use TensorRT for efficient inference of models such as YOLO and ResNet.
Comparison with NPU
NPU is a dedicated AI chip with high energy efficiency and controllable cost; TensorRT runs on NVIDIA GPUs, with a mature ecosystem and broad support. Industrial edge AI selection: NVIDIA Jetson + TensorRT is suitable for scenarios requiring strong compatibility and multiple models; NPU is suitable for large-scale, cost-sensitive scenarios. Both can be deployed with ONNX models.
Use Cases
Industrial machine vision for defect detection, object detection, and classification; security facial recognition and behavior analysis; autonomous driving perception. Edge AI gateways using Jetson + TensorRT can run complex models locally, reducing latency and bandwidth usage.
Build-Time Optimization and a Locked Engine
Layers are fused and kernels are chosen at build time
Inference optimization happens at build time. It is not decided afresh on every recognition. The builder reads the network, merges neighboring convolution, bias, and activation into fewer kernels, tries faster implementations for the architecture of the target graphics processor, and writes an engine. The engine holds both the fused weights and the schedule for that particular processor. The same network should therefore be built separately for different graphics processors. The precision chosen at build time decides the kernel. 32-bit floating point, 16-bit floating point, and 8-bit integer are not the same execution path. Eight-bit also needs a calibration table that says how each layer scales floating point down to integers.
Dynamic shapes can leave several profiles at build time, but the profiles are chosen in advance. They are not infinitely arbitrary. If a runtime input falls outside the profiles, the engine refuses it or falls onto a very slow path. Change the batch or the resolution and the best kernel changes too. A latency measured at one resolution cannot be extrapolated.
The engine is bound to a specific GPU and precision
The conditions for use are a graphics-processor model, a driver, and a builder version that match the site, and input preprocessing that matches what the build assumed. An engine file copied onto a graphics processor of another generation usually cannot be deserialized. Even if it loads, latency and results should not be assumed identical. Graphics memory has to hold the engine, the input buffer, and post-processing at the same time. If an industrial camera is decoded on the central processor and then copied to the graphics processor, the copy spends the time the fused kernels saved. Confirm the whole pipeline. Do not look only at the microseconds inside the inference kernel.
It suits a model that is already frozen, a device that is one known accelerator card, and a latency that has to fit inside the line takt. If the model structure changes every week, the cost of building and calibrating exceeds the benefit. In that case make the build a repeatable step, and archive the input sample, the precision, and the version number together, so devices on site do not each run an engine of unknown origin.
Calibration set and dynamic shape
Eight-bit calibration fails silently more easily than anything else. If the calibration pictures all come from a laboratory with normal exposure, backlight, oil film, and motion blur on site push activations outside the calibrated scales. Detection-box confidence falls as a whole, and the program reports no error. Calibrate with pictures taken on site, and compare outputs on the same batch of pictures before and after quantization. Leave layers whose difference is large at a higher precision. Another error is a shape optimization enabled at build time while runtime sends a side length that is not aligned. The engine refuses it outright. On the line the camera looks healthy and there is no result.
Version lock matters just as much. After a driver upgrade an old engine may fail to load. If the upgrade window did not keep the previous engine and the matching driver, the device is stuck with no way back. When several models share one card, limit concurrent contexts, or graphics-memory fragmentation makes the second engine fail to load. Finally, do not promise an embedded module the latency measured on a desktop card. Memory bandwidth and the power wall differ. Measure takt again on the target module, including the steady state after warmup, not the first inference.