isaac_ros_ess

Source code on GitHub.

Quickstart

1. Set up your development environment by following the instructions here.

2. Clone isaac_ros_common and this repository under ${ISAAC_ROS_WS}/src.

   cd ${ISAAC_ROS_WS}/src
   

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common.git
   

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_stereo_depth.git
   

3. Pull down a ROS Bag of sample data:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_dnn_stereo_depth && \
   git lfs pull -X "" -I "resources/rosbags/ess_rosbag"
   

4. Launch the Docker container using the run_dev.sh script:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && ./scripts/run_dev.sh
   

5. Install this package’s dependencies.

sudo apt-get install -y ros-humble-isaac-ros-ess ros-humble-isaac-ros-test

6. Download the pre-trained ESS model from the ESS model page:

   For ESS:

   cd /workspaces/isaac_ros-dev/src/isaac_ros_dnn_stereo_depth/resources && \
   wget 'https://api.ngc.nvidia.com/v2/models/nvidia/isaac/dnn_stereo_disparity/versions/3.0.0/files/ess.etlt'
   

   For Light ESS:

   cd /workspaces/isaac_ros-dev/src/isaac_ros_dnn_stereo_depth/resources && \
   wget 'https://api.ngc.nvidia.com/v2/models/nvidia/isaac/dnn_stereo_disparity/versions/3.0.0/files/light_ess.etlt'
   

7. Convert the encrypted model (.etlt) to a TensorRT engine plan:

   For ESS:

   /opt/nvidia/tao/tao-converter -k ess -t fp16 -e /workspaces/isaac_ros-dev/src/isaac_ros_dnn_stereo_depth/resources/ess.engine -o output_left,output_conf /workspaces/isaac_ros-dev/src/isaac_ros_dnn_stereo_depth/resources/ess.etlt
   

   For Light ESS:

   /opt/nvidia/tao/tao-converter -k ess -t fp16 -e /workspaces/isaac_ros-dev/src/isaac_ros_dnn_stereo_depth/resources/ess.engine -o output_left,output_conf /workspaces/isaac_ros-dev/src/isaac_ros_dnn_stereo_depth/resources/light_ess.etlt
   

8. Launch the ESS Disparity Node:

   To run ESS at a threshold of 0.9 run:

   ros2 launch isaac_ros_ess isaac_ros_ess.launch.py engine_file_path:=/workspaces/isaac_ros-dev/src/isaac_ros_dnn_stereo_depth/resources/ess.engine threshold:=0.9
   

   To run ESS at a threshold of 0.0 run:

   ros2 launch isaac_ros_ess isaac_ros_ess.launch.py engine_file_path:=/workspaces/isaac_ros-dev/src/isaac_ros_dnn_stereo_depth/resources/ess.engine threshold:=0.0
   

9. Open a second terminal and attach to the container:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
     ./scripts/run_dev.sh
   

10. In the second terminal, visualize and validate the output of the package:

    ros2 run isaac_ros_ess isaac_ros_ess_visualizer.py --enable_rosbag
    

    With ESS and threshold set to 0.9, the expected result is:

    

    With ESS and threshold set to 0.0, the expected result is:

    

Try More Examples

To continue your exploration, check out the following suggested examples:

• Tutorial with RealSense
• Tutorial with Zed
• Tutorial with Isaac Sim
• Tutorial with NITROS Graph
• Visualize ESS Output

Troubleshooting

Package not found while launching the visualizer script

Symptom

$ ros2 run isaac_ros_ess isaac_ros_ess_visualizer.py
Package 'isaac_ros_ess' not found

Solution

Use the colcon build --packages-up-to isaac_ros_ess command to build isaac_ros_ess; do not use the --symlink-install option. Run source install/setup.bash after the build.

Problem reserving CacheChange in reader

Symptom

When using a ROS bag as input, isaac_ros_ess throws an error if the input topics are published too fast:

[component_container-1] 2022-06-24 09:04:43.584 [RTPS_MSG_IN Error] (ID:281473268431152) Problem reserving CacheChange in reader: 01.0f.cd.10.ab.f2.65.b6.01.00.00.00|0.0.20.4 -> Function processDataMsg

Solution

Make sure that the ROS bag has a reasonable size and publish rate.

API

Overview

The isaac_ros_ess package offers functionality to generate a stereo disparity map from stereo images using a trained ESS model. Given a pair of stereo input images, the package generates a continuous disparity image for the left input image.

Usage

ros2 launch isaac_ros_ess isaac_ros_ess.launch.py engine_file_path:=<your ESS engine plan absolute path>

ESSDisparityNode

ROS Parameters

ROS Parameter	Type	Default	Description
engine_file_path	string	N/A - Required	The absolute path to the ESS engine file.
threshold	double	0.9	Threshold value ranges between 0.0 and 1.0 for filtering disparity with confidence. Pixels with confidence less than threshold will be marked as invalid in the disparity output. Value 0.0 means a fully dense disparity output.
image_type	string	"RGB_U8".	The input image encoding type. Supports "RGB_U8" and "BGR_U8". Note that if this parameter is not specified and there is an upstream Isaac ROS NITROS node, the type will be decided automatically.

ROS Topics Subscribed

ROS Topic	Interface	Description
left/image_rect	sensor_msgs/Image	The left image of a stereo pair.
right/image_rect	sensor_msgs/Image	The right image of a stereo pair.
left/camera_info	sensor_msgs/CameraInfo	The left camera model.
right/camera_info	sensor_msgs/CameraInfo	The right camera model.

Note

The images on input topics (left/image_rect and right/image_rect) should be a color image either in rgb8 or bgr8 format.

ROS Topics Published

ROS Topic	Interface	Description
disparity	stereo_msgs/DisparityImage	The continuous stereo disparity estimation.

Input Restrictions

1. The input left and right images must have the same dimension and resolution, and the resolution must be no larger than 1920x1200.

2. Each input pair (left/image_rect, right/image_rect, left/camera_info and right/camera_info) should have the same timestamp; otherwise, the synchronizing module inside the ESS Disparity Node will drop the input with smaller timestamps.

Output Interpretations

1. The isaac_ros_ess package outputs a disparity image with dimension same as the ESS model output dimension.

   ESS Model	Output Dimension
   ess.etlt	960 x 576
   light_ess.etlt	480 x 288

   The input images are rescaled to the ESS model input dimension before inferencing. There are two outputs from the ESS model with the same dimension: disparity output and confidence output. The disparity is filtered with confidence using a pre-configured threshold. Pixels with confidence less than the threshold is replaced with -1.0 as invalid before the inference result is published. For fully dense disparity output without confidence thresholding, set the threshold to 0.0.

2. The left and right CameraInfo are used to composite a stereo_msgs/DisparityImage. If you only care about the disparity image, and don’t need the baseline and focal length information, you can pass dummy camera messages.