Unlocking the Full Potential of Edge AI: NVIDIA JetPack 6.2 and Super Mode
Summary
NVIDIA has taken a significant leap forward in enhancing AI capabilities on edge devices with the release of JetPack 6.2. This update introduces the Super Mode feature to the Jetson Orin Nano and Jetson Orin NX modules, promising up to a twofold increase in AI performance. This article delves into the details of Super Mode, its benefits, and how it revolutionizes edge AI applications.
Introduction to Super Mode
Super Mode is a new feature in JetPack 6.2 that unlocks higher operational frequencies for CPU, GPU, and other processing units on the Jetson Orin modules. This results in unprecedented AI model performance, particularly beneficial for large language models (LLMs), vision language models (VLMs), and vision transformers (ViTs).
Enhanced Performance with Super Mode
The Jetson Orin Nano and Jetson Orin NX modules see significant performance boosts with Super Mode. For instance, the Jetson Orin Nano 8GB module shows a performance increase from 14 to 19.1 tokens per second on the Llama 3.1 8B model. Similarly, VLMs like the VILA 1.5 3B on the Orin NX 16GB module see an increase from 1 to 1.3 tokens per second.
New Power Modes
JetPack 6.2 introduces new reference power modes for the Jetson Orin Nano and Jetson Orin NX series. These include a 25W mode and an uncapped MAXN SUPER mode for the Jetson Orin Nano, while the Orin NX series benefits from a new 40W power mode. Here is a breakdown of the new power modes:
| Module | Existing Reference Power Modes | New Power Modes |
|---|---|---|
| NVIDIA Jetson Orin Nano 4GB | 7W, 10W | 10W, 25W, MAXN SUPER |
| NVIDIA Jetson Orin Nano 8GB | 7W, 15W | 15W, 25W, MAXN SUPER |
| NVIDIA Jetson Orin NX 8GB | 10W, 15W, 20W, MAXN | 10W, 15W, 20W, 40W, MAXN SUPER |
| NVIDIA Jetson Orin NX 16GB | 10W, 15W, 25W, MAXN | 10W, 15W, 25W, 40W, MAXN SUPER |
Detailed Specifications
The following table compares the detailed specifications of Jetson Orin Nano and Jetson Orin NX modules in their original and Super Mode configurations:
| Module | Peak AI Performance INT8 | NVIDIA Ampere GPU | CPU | DLA (S/D) | DRAM BW | Reference Module Power |
|---|---|---|---|---|---|---|
| Orin Nano 4GB | 20 TOPS (Sparse) 10 TOPS (Dense) | 512 CUDA Cores 16 Tensor Cores 625 MHz | 6X A78 1.5 GHz | NA | 34 GB/s | 7W, 10W |
| Orin Nano 4GB (Super) | 34 TOPS (Sparse) 17 TOPS (Dense) | 512 CUDA Cores 16 Tensor Cores 1020 MHz | 6X A78 1.7 GHz | NA | 51 GB/s | 10W, 25W, MAXN SUPER |
| Orin Nano 8GB | 40 TOPS (Sparse) 20 TOPS (Dense) | 1024 CUDA Cores 32 Tensor Cores 625 MHz | 6X A78 1.5 GHz | NA | 68 GB/s | 7W, 15W |
| Orin Nano 8GB (Super) | 67 TOPS (Sparse) 33 TOPS (Dense) | 1024 CUDA Cores 32 Tensor Cores 1020 MHz | 6X A78 1.7 GHz | NA | 102 GB/s | 15W, 25W, MAXN SUPER |
| Orin NX 8GB | 50 TOPS (Sparse) 25 TOPS (Dense) | 1024 CUDA Cores 32 Tensor Cores 765 MHz | 6X A78 2.0 GHz | 20/10 INT8 TOPs | 102 GB/s | 10W, 15W, 20W, MAXN |
| Orin NX 8GB (Super) | 77 TOPS (Sparse) 38 TOPS (Dense) | 1024 CUDA Cores 32 Tensor Cores 1173 MHz | 6X A78 2.0 GHz | 40/20 INT8 TOPs | 102 GB/s | 10W, 15W, 20W, 40W, MAXN SUPER |
| Orin NX 16GB | 60 TOPS (Sparse) 30 TOPS (Dense) | 1024 CUDA Cores 32 Tensor Cores 918 MHz | 8X A78 2.0 GHz | 40/20 INT8 TOPs | 102 GB/s | 10W, 15W, 25W, MAXN |
| Orin NX 16GB (Super) | 77 TOPS (Sparse) 38 TOPS (Dense) | 1024 CUDA Cores 32 Tensor Cores 1173 MHz | 8X A78 2.0 GHz | 80/40 INT8 TOPs | 102 GB/s | 10W, 15W, 25W, 40W, MAXN SUPER |
Power Estimator Tool
NVIDIA provides the Power Estimator Tool to help create custom power profiles and nvpmodel configuration files. This tool has been updated with Super Mode and is essential for finding the right balance between power consumption and performance.
Conclusion
NVIDIA JetPack 6.2’s Super Mode is a game-changer for edge AI applications. By unlocking higher operational frequencies, it significantly boosts AI performance on the Jetson Orin Nano and Jetson Orin NX modules. With new power modes and detailed specifications, developers can now achieve unprecedented AI model performance. The Power Estimator Tool further aids in optimizing power consumption and performance. This update sets a new benchmark for generative AI on small-scale devices, making it an exciting time for AI innovation.