Ultimately, he and his group want to help people design more accurate and robust deep-learning models that are better suited for their intended application. “We used to debug systems just by looking at their parameters and figuring out whether they make sense or not. Now, we need ‘mind-reading tools’ that can take this really complicated

NVIDIA adds Python support for edge AI inference on Jetson devices
NVIDIA has detailed how developers can run AI inference directly from Python on Jetson AGX Orin devices using its CUDA Deep Learning Accelerator (cuDLA) bindings. The approach allows applications to manage inference workloads on dedicated hardware without requiring C++ code to control the runtime.
The cuda.bindings.cudla module, introduced in cuda-bindings 13.3.0, provides Python access to NVIDIA’s existing cuDLA C++ API. It supports applications that combine GPU processing with Deep Learning Accelerator (DLA) inference on local hardware.
According to NVIDIA, DLA executes supported neural network operations separately from the GPU, freeing GPU capacity for other application tasks. The dedicated accelerator can also improve power efficiency when running inference on devices with limited computing resources.
Running AI inference through Python
The DLA is a dedicated accelerator designed to execute neural network operations such as convolutions and pooling. Unlike a GPU, which supports a broader range of computations, the DLA is limited to specific operations supported by its hardware.
NVIDIA’s Python bindings provide access to the cuDLA runtime through a Cython-generated interface. Developers can use the module to create DLA devices, load compiled neural networks, manage memory buffers, and submit inference tasks without writing C++ runtime code.
In hybrid mode, developers use CUDA streams to coordinate GPU and DLA operations within the same application. GPU-based preprocessing, DLA inference, and GPU-based postprocessing can be scheduled in sequence, while the neural network executes directly on DLA hardware.
The Python implementation currently supports hybrid execution but not standalone DLA mode. NVIDIA said standalone execution requires additional buffer-sharing and synchronisation mechanisms that are not available through the new bindings.
Before running inference, developers must compile trained neural networks into a format supported by DLA hardware. NVIDIA’s workflow uses TensorRT’s trtexec tool to convert an ONNX model into a standalone DLA loadable, which contains the instructions needed for execution on the accelerator.
Once compiled, the loadable can be executed through the Python bindings without requiring the TensorRT runtime during inference. The underlying libcudla.so library is included with NVIDIA’s JetPack software, removing the need for a separate cuDLA software development kit on compatible Jetson Orin devices.
GPU performance and model limitations
NVIDIA previously reported that DLA delivered three to five times better performance per watt than GPU execution for selected inference workloads on Jetson Orin. The results, published in August 2023, varied depending on the model and hardware power mode.
On the Jetson AGX Orin 64GB, NVIDIA reported theoretical peak sparse INT8 performance of 90 TOPS across two DLAs, compared with 41 TOPS from the GPU, under the platform’s 30W power mode. These figures represent theoretical computing capacity rather than measured application inference speeds.
The November 2026 technical walkthrough does not provide updated comparisons of inference latency, throughput, or power consumption using the Python bindings.
NVIDIA’s TensorRT documentation identifies unsupported layers, parameter restrictions, and hardware resource limits as reasons certain operations cannot run on DLA. On compatible Jetson configurations, TensorRT supports GPU fallback, allowing incompatible operations to execute on the GPU when that option is enabled during engine compilation.
Without fallback, unsupported operations can prevent compilation. NVIDIA’s Python example uses a standalone DLA loadable rather than a TensorRT engine configured for GPU fallback, requiring the compiled operations to be compatible with DLA.
NVIDIA also notes that developers sometimes need to modify their ONNX models to remove incompatible operations before generating a DLA loadable. Its Python example uses an FP16 model with fixed input dimensions.
Input dimensions present another restriction. NVIDIA’s documentation states that DLA does not support dynamic dimensions, requiring the minimum, maximum, and optimal dimensions in a TensorRT optimisation profile to match.
By comparison, GPU-based TensorRT execution supports dynamic shapes through optimisation profiles. DLA models must use dimensions that match those specified during compilation, limiting how variable-sized inputs can be handled without preparing additional model configurations.
Hardware support and deployment requirements
Hardware compatibility also affects where developers can deploy DLA inference workloads. According to NVIDIA’s TensorRT 11.4.1 documentation, DLA is supported on Jetson AGX Orin running JetPack 7.2.2 or later, but is not supported on Jetson Thor or NVIDIA DriveOS deployments in that release.
NVIDIA has also extended the cuDLA Python interface to RTX Spark, providing access to DLA inference on supported Windows on Arm laptops. TensorRT 11.4.1 lists Windows on Arm DLA support as being in beta, with a more restricted set of supported neural network layers than Jetson.
Unlike the supported Jetson configuration, Windows on Arm does not provide GPU fallback. Every layer in a DLA network must be compatible with the accelerator, or the model will fail to compile.
Windows on Arm also has a batch-size limit of one, while the supported Jetson implementation allows larger batch sizes within the accelerator’s documented restrictions. NVIDIA’s documentation further states that DLA engines built during the Windows on Arm beta will need to be rebuilt when support reaches general availability.
Model precision requirements also differ between the platforms. On Jetson AGX Orin, DLA supports FP16 and INT8 computation, subject to tensor format and memory alignment restrictions. Windows on Arm uses FP16 computation and does not support pure INT8 inference, although certain quantisation formats are supported.
NVIDIA has published sample code covering model compilation, memory management, and inference execution. The CUDA Python repository also includes tests that can validate the cuDLA API interface without requiring DLA hardware.
Developers can access the bindings through the cuda-bindings Python package, starting with version 13.3.0. NVIDIA’s published example uses TensorRT 11.4.1, CUDA 13.4 Update 1, and JetPack 7.2.2 or later on Jetson AGX Orin.
(Photo by Florian Olivo)
See also: NVIDIA Minerva targets AI factory observability
Want to dive deeper into the tools and frameworks shaping modern development? Check out the AI & Big Data Expo, taking place in Amsterdam, California, and London. Explore cutting-edge sessions on machine learning, data pipelines, and next-gen AI applications. The event is part of TechEx and co-located with other leading technology events. Click here for more information.
Developer Tech News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.
