Perceiving and understanding the world around them is a critical challenge for autonomous robots.
In conjunction with ROS World 2021, NVIDIA announced its latest efforts to deliver performant perception technologies to the ROS developer community. These initiatives will accelerate product development, improve product performance and ultimately simplify the task of incorporating cutting-edge computer vision and AI/ML functionality into ROS-based robotic applications.
Announcement Highlights:
- Highest Performing Real-Time Stereo Odometry solution available as a ROS package
- All NVIDIA Inference DNNs available on NGC available as a ROS package with examples for image segmentation and pose estimation
- New Synthetic Data Generation (SDG) workflow in NVIDIA Isaac Sim to create production-quality datasets at scale for Vision AI training
- Isaac Sim on Omniverse with out-of-the box support for ROS is now available with the most developer-friendly release to date
Isaac ROS GEMs — Optimized Performance
Figure 1. Block diagram representing software components in Isaac ROS stack showing the new Isaac ROS GEMs.
Isaac ROS GEMs provide packages that encompass image processing and computer vision, including DNN-based algorithms highly optimized for NVIDIA GPUs and the full Jetson line up.
Highlighted GEM: Stereo Visual Odometry — Best In Class Accuracy and Optimized Performance
As autonomous machines move around in their environments they must keep track of where they are. Visual odometry solves this problem by estimating where a camera is relative to its starting position. The Isaac ROS GEM for Stereo Visual Odometry provides this powerful functionality to ROS developers.
This GEM offers the best accuracy for a real-time (>60fps @720p) stereo camera visual odometry solution. Publicly available results based on the widely used KITTI database can be referenced here. In addition to being very accurate, this GPU accelerated package runs extremely fast. In fact, it is now possible to run HD resolution in real-time on a Jetson Xavier AGX.
Highlighted GEM: DNN Inference — All NVIDIA NGC DNN Inference models available to ROS Developers
DNN Inference GEM is a set of ROS2 packages that allow developers to use any of NVIDIA’s numerous inference models available on NGC or even provide their own DNN. Further tuning of pre-trained models or optimizations of developers’ own custom models can be done with the NVIDIA TAO Toolkit.
After optimization, these packages are deployed by TensorRT or Triton, NVIDIA’s inference server. Optimal inference performance will be achieved with the nodes leveraging TensorRT, NVIDIA’s high performance inference SDK. If the desired DNN model isn’t supported by TensorRT then Triton should be used to deploy the model. The GEM includes native support for U-Net and DOPE. The U-Net package, based on TensorRT, can be used for generating semantic segmentation masks from images. And the DOPE package can be used for 3D pose estimation for all detected objects.
This tool is the fastest way to incorporate performant AI inference in a ROS application.
Figure 2. A composite image from three different image processing algorithms using Isaac ROS GEMs — DNN [PeopleSemSegnet/AprilTags/Disparity(Depth)]
Isaac Sim GA Release
The GA release of Isaac Sim will be available in November 2021 as the most developer friendly release to date. With numerous improvements in the UI, performance, and useful building blocks it will lead to better simulations getting built much faster. In addition, an improved ROS bridge and more ROS samples will enhance the developer experience for ROS developers.
New For This Release (2021.2 Release scheduled for November 2021)
- Improved performance, reduced memory usage and startup times
- Improved Occupancy Map Generation, URDF Importer
- New Environments: Large warehouse, Office, Hospital
- New Python building blocks for interfacing with robots, objects, environments
- Improved performance for ROS/ROS2 Bridge, Depth point cloud, Lidar point cloud
- Sample Updates
- Multi-robot navigation with ROS2
- SDG with Domain Randomization in Jupyter
Video: Demo of Joint Control of Franka using ROS MoveIt in Isaac Sim:
New Synthetic Data Generation Workflow — Production Datasets from Isaac Sim
An autonomous robot requires large and diverse datasets to train the numerous AI models running its perception stack. Getting all of this training data from real world scenarios is cost prohibitive and in the corner cases, potentially dangerous. The new synthetic data workflow provided with Isaac Sim is designed to build production quality datasets that address the safety and quality concerns of autonomous robots.
The developer building the datasets controls the stochastic distribution of the objects in the scene, the scene itself, the lighting, and the synthetic sensors. The developer also has the fine grained control as well to ensure that important corner cases are included in the datasets. Finally, this workflow supports versioning and debugging information so that datasets can be exactly reproduced for auditing and safety purposes.
Figure 3. Examples of Synthetic Data from the Sensors library in Isaac Sim
Getting Started
Developers interested in getting started with Isaac ROS should learn more here.
Be sure to stop by NVIDIA’s virtual booth at ROS World 21, attend our NVIDIA ROS roundtable, and watch the technical presentation on NVIDIA Isaac Sim.
- On Oct. 21 at 4pm CT, join Hammad Mazhar, NVIDIA Lead Simulation Engineer to learn how NVIDIA Isaac Sim can be used to simulate different ROS/ROS2-driven workflows.
- Join the “Simulation Tools on ROS” panel on Oct. 21 at 5:20pm CT where Liila Torabi, NVIDIA Isaac Senior Product Manager, and developers of some of the world’s largest robotics simulation products will discuss the past, present, and future of simulation.
View robotics applications in action: