Summary: Real-time surgical guidance is crucial for minimally invasive and robotic-assisted surgeries. Combining preoperative 3D imaging with intra-operative video can significantly improve surgical workflows. NVIDIA Holoscan, a platform for real-time sensor processing, AI, and I/O, has been integrated with ImFusion’s proprietary SDK to create a system that can fuse preoperative and intra-operative data in real-time. This system enhances surgical accuracy, reduces complications, and improves outcomes.

Real-Time Surgical Guidance: A New Frontier in Minimally Invasive Surgery

Minimally invasive and robotic-assisted surgeries have revolutionized the way surgeons perform complex procedures. However, these surgeries require accurate navigation and detailed understanding of the patient’s anatomy. To address this challenge, developers in the fields of image-guided surgery and surgical vision are working on creating systems that can combine preoperative 3D imaging with intra-operative video in real-time.

The Challenge of Combining Multi-Modal Imaging Data

During preoperative planning, surgeons rely on multi-modal imaging techniques, including 3D diagnostic imaging modalities such as computed tomography (CT) scans, to identify abnormalities, designate target zones, and pinpoint critical structures such as blood vessels. However, combining these preoperative 3D image datasets with intra-operative video seamlessly during live surgical procedures remains a significant challenge.

NVIDIA Holoscan: A Platform for Real-Time Sensor Processing

NVIDIA Holoscan is a platform that enables real-time sensor processing, AI, and I/O. It has been integrated with ImFusion’s proprietary SDK to create a system that can fuse preoperative and intra-operative data in real-time. This system tracks the surface of a targeted anatomical structure in the form of a 3D mesh—a digital model that accurately depicts the structure’s shape and contours—and blends it smoothly into the surgeon’s view.

ImFusion’s Solution: Real-Time 3D Surgical Guidance

ImFusion’s solution is built on their proprietary ImFusion SDK, which bundles algorithms for image processing, registration, analysis, and visualization. Integrating NVIDIA Holoscan into the ImFusion SDK unlocked new levels of performance, efficiency, and flexibility. The system achieves a median frame rate of ~13.5 Hz and an end-to-end latency below 75ms, enabling surgeons to receive instantaneous visual feedback and providing an unprecedented view of the surgical scene.

Key Neural Networks: Stereo Depth Estimation, Optical Flow Estimation, and Segmentation

At the core of ImFusion’s system are three key neural networks:

  • Stereo Depth Estimation: This network generates depth information from endoscopic stereo video frames, using a state-of-the-art CNN-based model trained on synthetic data.
  • Optical Flow Estimation: This network calculates 2D pixel displacements between frames, ensuring robust performance across various surgical scenarios.
  • Segmentation: This deep learning segmentation model identifies surgical instruments and target tissue, crucial for accurate tracking and overlay.

Enhanced Surgical Navigation and Accelerated Development

The integration of NVIDIA Holoscan with ImFusion’s SDK accelerates the development of AI-enhanced software as a medical device (SaMD). This collaboration between ImFusion and NVIDIA Holoscan is upleveling the art of the possible in minimally-invasive and robotic-assisted procedures, combining AI, accelerated computing, and domain specificity to enhance precision, performance, and safety.

Ecosystem Collaboration and Open-Source Contributions

ImFusion’s contributions to Holoscan reference applications can be integrated and built upon by its medtech customers. The company invites developers to explore and contribute to the Holoscan reference application repository to expand the ecosystem, accelerate the development of AI-enhanced medical devices, and advance real-time sensor fusion for surgical guidance.

Clinical Relevance and Future Directions

The ORSI Academy, Europe’s largest robotic surgery training center, has partnered with both NVIDIA and ImFusion to guide the development and strengthen its clinical relevance. Dr. Pieter De Backer, engineer and surgical resident leading Orsi Innotech, the surgical AI department of ORSI Academy, emphasizes the importance of seamlessly blending live video feeds with overlaid 3D mesh projections to enhance surgical navigation during minimally-invasive or robotic-assisted procedures.

Conclusion

Real-time surgical guidance by fusing multi-modal imaging with NVIDIA Holoscan is a groundbreaking advancement in minimally invasive and robotic-assisted surgeries. This technology has the potential to significantly improve surgical workflows, enhance surgical accuracy, reduce complications, and improve outcomes. As the medical community continues to embrace AI-enhanced medical devices, the collaboration between ImFusion and NVIDIA Holoscan sets a new standard for real-time sensor fusion in surgical guidance.