Unlocking Next-Gen Vision AI: Deep Dive into NVIDIA DeepStream 7.0
Summary: NVIDIA DeepStream 7.0 is a milestone release that empowers developers to build next-generation vision AI applications. This article delves into the key features and enhancements of DeepStream 7.0, including its support for sensor fusion, improved 3D tracking, and streamlined AI pipeline optimization.
DeepStream 7.0: A New Era for Vision AI
NVIDIA DeepStream is a powerful SDK that provides GPU-accelerated building blocks for end-to-end vision AI pipelines. With over 40 plugins available off-the-shelf, developers can deploy fully optimized pipelines with cutting-edge AI inference, object tracking, and seamless integration with popular IoT message brokers.
Key Features of DeepStream 7.0
- Sensor Fusion with BEVFusion: DeepStream 7.0 supports BEVFusion, a sensor fusion model that integrates LIDAR and radar inputs with camera inputs, enhancing environmental perception solutions.
- Improved 3D Tracking: The release includes enhanced Single-View 3D tracking and support for the NvDCF tracker with PVA backend on Jetson.
- Streamlined AI Pipeline Optimization: PipeTuner, a new tool in DeepStream 7.0, automates AI pipeline optimization using annotated data and user videos.
- Python APIs: New DeepStream libraries are available in Python, making it easier to integrate into custom frameworks.
- Service Maker Framework: A new application layer that removes the need to understand GStreamer application programming paradigm.
- WSL2 Support: Developers can now use Windows Subsystem for Linux (WSL2), eliminating the need for a dedicated Linux box.
Enhanced REST API Support
DeepStream 7.0 offers enhanced REST API support to control AI pipelines on-the-fly, whether deployed at the far edge or the cloud. This feature provides greater flexibility and ease of use for developers.
Performance Optimizations
The release includes various performance optimizations, such as improved ReID accuracy in trackers, enhanced fault-tolerance of sub-batching for trackers, and new GStreamer plugins like nvimagedec and nvimageenc based on CUDA nvjpeg APIs.
Integration with NVIDIA TAO Toolkit
DeepStream 7.0 integrates models from the NVIDIA TAO toolkit (previously called NVIDIA Transfer Learning Toolkit), providing continued support for 2D body pose estimation, facial landmark estimation, emotion recognition, gaze, heart rate, and gesture.
New Samples and Plugins
The release includes new samples like deepstream-ipc-test, which demonstrates decoder buffer sharing IPC use-case on Jetson platforms for live streams to optimize NVDEC HW utilization. Additionally, there are new plugins such as Trafficcamnet as a primary detector and two new secondary classifier models (VehicleMakeNet, VehicleTypeNet).
Comparison of Key Features
| Feature | Description |
|---|---|
| Sensor Fusion | Supports BEVFusion with LIDAR, radar, and camera inputs. |
| 3D Tracking | Enhanced Single-View 3D tracking and NvDCF tracker support. |
| AI Pipeline Optimization | Automated optimization with PipeTuner. |
| Python APIs | New DeepStream libraries in Python for easier integration. |
| Service Maker | Simplifies application development by removing GStreamer complexity. |
| WSL2 Support | Eliminates the need for a dedicated Linux box. |
Technical Specifications
| Specification | Detail |
|---|---|
| Supported Platforms | x86/dGPU, Jetson, ARM SBSA, IGX |
| Triton Support | Triton 23.10 for x86/dGPU, Triton 24.03 for Jetson |
| GStreamer Plugins | nvimagedec, nvimageenc, nvstreammux |
| Encoder Support | AV1 encoder support added |
| REST API | Enhanced support for controlling AI pipelines on-the-fly |
Conclusion
NVIDIA DeepStream 7.0 is a significant milestone in the development of next-generation vision AI applications. With its support for sensor fusion, improved 3D tracking, and streamlined AI pipeline optimization, developers can now build more sophisticated and efficient vision AI pipelines. The release’s focus on ease of use, with features like Python APIs and the Service Maker framework, makes it accessible to a broader range of developers. As vision AI continues to evolve, DeepStream 7.0 provides the tools necessary to unlock its full potential.