isaac_ros_stereo_image_proc

Source code available on GitHub.

Quickstart

Set Up Development Environment

1. Set up your development environment by following the instructions in getting started.

2. (Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

   Note

   We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the development environment during installation, which will interrupt the quickstart process.

Download Quickstart Assets

1. Download quickstart data from NGC:

   Make sure required libraries are installed.

   sudo apt-get install -y curl jq tar
   

   Then, run these commands to download the asset from NGC:

   NGC_ORG="nvidia"
   NGC_TEAM="isaac"
   PACKAGE_NAME="isaac_ros_stereo_image_proc"
   NGC_RESOURCE="isaac_ros_stereo_image_proc_assets"
   NGC_FILENAME="quickstart.tar.gz"
   MAJOR_VERSION=4
   MINOR_VERSION=0
   VERSION_REQ_URL="https://catalog.ngc.nvidia.com/api/resources/versions?orgName=$NGC_ORG&teamName=$NGC_TEAM&name=$NGC_RESOURCE&isPublic=true&pageNumber=0&pageSize=100&sortOrder=CREATED_DATE_DESC"
   AVAILABLE_VERSIONS=$(curl -s \
       -H "Accept: application/json" "$VERSION_REQ_URL")
   LATEST_VERSION_ID=$(echo $AVAILABLE_VERSIONS | jq -r "
       .recipeVersions[]
       | .versionId as \$v
       | \$v | select(test(\"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+$\"))
       | split(\".\") | {major: .[0]|tonumber, minor: .[1]|tonumber, patch: .[2]|tonumber}
       | select(.major == $MAJOR_VERSION and .minor <= $MINOR_VERSION)
       | \$v
       " | sort -V | tail -n 1
   )
   if [ -z "$LATEST_VERSION_ID" ]; then
       echo "No corresponding version found for Isaac ROS $MAJOR_VERSION.$MINOR_VERSION"
       echo "Found versions:"
       echo $AVAILABLE_VERSIONS | jq -r '.recipeVersions[].versionId'
   else
       mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets && \
       FILE_REQ_URL="https://api.ngc.nvidia.com/v2/resources/$NGC_ORG/$NGC_TEAM/$NGC_RESOURCE/\
   versions/$LATEST_VERSION_ID/files/$NGC_FILENAME" && \
       curl -LO --request GET "${FILE_REQ_URL}" && \
       tar -xf ${NGC_FILENAME} -C ${ISAAC_ROS_WS}/isaac_ros_assets && \
       rm ${NGC_FILENAME}
   fi
   

Build isaac_ros_stereo_image_proc

Binary PackageBuild from Source

1. Activate the Isaac ROS environment:

   isaac-ros activate
   

2. Install the prebuilt Debian package:

   sudo apt-get update
   

   sudo apt-get install -y  ros-jazzy-isaac-ros-image-proc ros-jazzy-isaac-ros-stereo-image-proc
   

Run Launch File

RealSense CameraZED Camera

1. Ensure that you have already set up your RealSense camera using the RealSense setup tutorial. If you have not, please set up the sensor and then restart this quickstart from the beginning.

2. Continuing inside the container, install the following dependencies:

   sudo apt-get update
   

   sudo apt-get install -y ros-jazzy-isaac-ros-examples ros-jazzy-isaac-ros-realsense ros-jazzy-isaac-ros-depth-image-proc ros-jazzy-isaac-ros-image-proc
   

3. Run the launch file, which launches the example, and wait for 10 seconds.

   ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_stereo_rect,disparity,disparity_to_depth,point_cloud_xyz
   

4. Observe point cloud output /points on a separate terminal with the command:

   ros2 topic echo /points
   

Try More Examples

To continue your exploration, check out the following suggested examples:

• Tutorial with Isaac Sim

API

Overview

The isaac_ros_stereo_image_proc package offers functionality for handling image pairs from a binocular/stereo camera setup, calculating the disparity between the two images, and producing a point cloud with depth information. It largely replaces the stereo_image_proc package.

Available Components

DisparityNode

ROS Parameters

ROS Parameter	Type	Default	Description
backend	string	CUDA	The VPI backend to use. Choices are: CUDA, JETSON.
max_disparity	float	256	The maximum disparity value per pixel. With JETSON backend, this value must be 128 or 256.
confidence_threshold	int	65023	Confidence threshold for the VPI SGM algorithm.
confidence_type	int	0	Confidence type used by the VPI SGM algorithm. Valid values: 0 (absolute), 1 (relative), 2 (inference).
window_size	int	7	Window size for SGM disparity calculation.
num_passes	int	2	Number of SGM passes to compute the result.
p1	int	8	Penalty on disparity changes of +/- 1 between neighboring pixels.
p2	int	120	Penalty on disparity changes of more than 1 between neighboring pixels.
p2_alpha	int	1	Alpha used to scale p2.
quality	int	1	Quality of disparity output.

ROS Topics Subscribed

ROS Topic	Interface	Description
left/image_rect	NitrosImage	Left rectified image.
left/camera_info	NitrosCameraInfo	Left camera info.
right/image_rect	NitrosImage	Right rectified image.
right/camera_info	NitrosCameraInfo	Right camera info.

ROS Topics Published

ROS Topic	Interface	Description
disparity	NitrosDisparityImage	Disparity image.

Note

DisparityNode with the JETSON backend requires a max_disparity value of 128 or 256. In addition, the JETSON backend requires nv12 input image format. You can use the ImageFormatConverterNode to convert the input to nv12.

Note

For optimal performance on NVIDIA Thor using the JETSON backend by configuring DisparityNode parameters to:

• backend: JETSON

• max_disparity: 256 (128 also supported depending on your range)

• confidence_threshold: 32767

• confidence_type: 2 (inference)

• window_size: 5

• num_passes: 3

• p1: 3

• p2: 48

• p2_alpha: 0

• quality: 6

PointCloudNode

ROS Parameters

ROS Parameter	Type	Default	Description
use_color	bool	false	Whether the output point cloud should be colorized.
unit_scaling	float	1.0	Scale applied to the XYZ values of the point cloud.
output_height	uint16_t	1200	Height used to size internal buffers for the output point cloud.
output_width	uint16_t	1920	Width used to size internal buffers for the output point cloud.

ROS Topics Subscribed

ROS Topic	Interface	Description
left/image_rect_color	NitrosImage	Left rectified image used for color.
left/camera_info	NitrosCameraInfo	Left camera info.
right/camera_info	NitrosCameraInfo	Right camera info.
disparity	NitrosDisparityImage	Disparity image input.

ROS Topics Published

ROS Topic	Interface	Description
points2	NitrosPointCloud	Output point cloud.

DisparityToDepthNode

ROS Parameters

This component has no ROS parameters.

ROS Topics Subscribed

ROS Topic	Interface	Description
disparity	NitrosDisparityImage	Disparity image input.

ROS Topics Published

ROS Topic	Interface	Description
depth	NitrosImage	Depth image output.