Isaac ROS DNN Stereo Depth
Overview
Isaac ROS DNN Stereo Depth provides a GPU-accelerated package for DNN-based
stereo disparity. Stereo disparity is calculated from a
time-synchronized image pair sourced from a stereo camera and is used to
produce a depth image or a point cloud for a scene. The isaac_ros_ess
package uses the ESS DNN
model
to perform stereo depth estimation via continuous disparity prediction.
Given a pair of stereo input images, the package generates a disparity
map of the left input image.
ESS is used in a graph of nodes to provide a disparity prediction from an input left and right stereo image pair. Images to ESS need to be rectified and resized to the appropriate input resolution. The aspect ratio of the image is recommended to be maintained, so the image may need to be cropped and resized to maintain the input aspect ratio. The graph for DNN encode, DNN inference, and DNN decode is included in the ESS node. Inference is performed using TensorRT, as the ESS DNN model is designed with optimizations supported by TensorRT. ESS node is agnostic to the model dimension and disparity output has the same dimension as the ESS model.
Quickstarts
Isaac ROS NITROS Acceleration
This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
ESS DNN
ESS stands for Efficient Semi-Supervised stereo disparity, developed by NVIDIA.
The ESS DNN is used to predict the disparity for each pixel from stereo camera image pairs.
This network has improvements over classic CV approaches that use epipolar geometry to compute disparity,
as the DNN can learn to predict disparity in cases where epipolar geometry feature matching fails.
The semi-supervised learning and stereo disparity matching makes the ESS DNN robust in environments unseen in
the training datasets and with occluded objects. This DNN is optimized for and
evaluated with color (RGB) global shutter stereo camera images, and accuracy may vary with
monochrome stereo images used in analytic computer vision approaches to stereo disparity.
The predicted disparity values
represent the distance a point moves from one image to the other
in a stereo image pair (a.k.a. the binocular image pair). The disparity is inversely proportional
to the depth (i.e. disparity = focalLength x baseline / depth). Given the
focal length and baseline
of the camera that generates a stereo image pair, the predicted disparity map from the isaac_ros_ess package
can be used to compute depth and generate a point cloud.
Note
Compare the requirements of your use case against the package input limitations.
DNN models
An ESS model is required to run isaac_ros_ess.
NGC provides pre-trained models for use in your robotics application.
ESS models are available on NGC, providing robust
depth estimation.
Click here for more information on how to use NGC models.
Confidence and Density
ESS DNN provides two outputs: disparity estimation and confidence estimation.
The disparity output can be filtered, by thresholding the confidence output,
to trade-off between confidence and density.
isaac_ros_ess filters out pixels with low confidence by setting:
disparity[confidence < threshold] = -1 # -1 means invalid
The choice of threshold value is dependent on use case.
Resolution and Performance
NGC Provides ESS 3.0 and Light ESS 3.0 models for trade-off between resolution and performance. A detailed comparison of the two models can be found here.
Model Name |
Disparity Resolution |
|---|---|
Estimate disparity at 1/4 HD resolution |
|
Estimate disparity at 1/16 HD resolution |
Packages
Supported Platforms
This package is designed and tested to be compatible with ROS 2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.
Note
Versions of ROS 2 earlier than Humble are not supported. This package depends on specific ROS 2 implementation features that were only introduced beginning with the Humble release.
Platform |
Hardware |
Software |
Notes |
|---|---|---|---|
Jetson |
For best performance, ensure that power settings are configured appropriately. |
||
x86_64 |
NVIDIA GPU |
Docker
To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.
Note
All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.
Customize your Dev Environment
To customize your development environment, reference this guide.
Updates
Date |
Changes |
|---|---|
2023-10-18 |
Updated for ESS 3.0 with confidence thresholding in multiple resolutions |
2023-05-25 |
Upgraded model (1.1.0) |
2023-04-05 |
Source available GXF extensions |
2022-10-19 |
Updated OSS licensing |
2022-08-31 |
Update to be compatible with JetPack 5.0.2 |
2022-06-30 |
Initial release |