isaac_ros_foundationstereo

Source code available on GitHub.

Quickstart

Set Up Development Environment

1. Set up your development environment by following the instructions in getting started.

2. (Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

   Note

   We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the development environment during installation, which will interrupt the quickstart process.

Download Quickstart Assets

1. Download quickstart data from NGC:

   Make sure required libraries are installed.

   sudo apt-get install -y curl jq tar
   

   Then, run these commands to download the asset from NGC:

   NGC_ORG="nvidia"
   NGC_TEAM="isaac"
   PACKAGE_NAME="isaac_ros_foundationstereo"
   NGC_RESOURCE="isaac_ros_foundationstereo_assets"
   NGC_FILENAME="quickstart.tar.gz"
   MAJOR_VERSION=4
   MINOR_VERSION=3
   VERSION_REQ_URL="https://catalog.ngc.nvidia.com/api/resources/versions?orgName=$NGC_ORG&teamName=$NGC_TEAM&name=$NGC_RESOURCE&isPublic=true&pageNumber=0&pageSize=100&sortOrder=CREATED_DATE_DESC"
   AVAILABLE_VERSIONS=$(curl -s \
       -H "Accept: application/json" "$VERSION_REQ_URL")
   LATEST_VERSION_ID=$(echo $AVAILABLE_VERSIONS | jq -r "
       .recipeVersions[]
       | .versionId as \$v
       | \$v | select(test(\"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+$\"))
       | split(\".\") | {major: .[0]|tonumber, minor: .[1]|tonumber, patch: .[2]|tonumber}
       | select(.major == $MAJOR_VERSION and .minor <= $MINOR_VERSION)
       | \$v
       " | sort -V | tail -n 1
   )
   if [ -z "$LATEST_VERSION_ID" ]; then
       echo "No corresponding version found for Isaac ROS $MAJOR_VERSION.$MINOR_VERSION"
       echo "Found versions:"
       echo $AVAILABLE_VERSIONS | jq -r '.recipeVersions[].versionId'
   else
       mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets && \
       FILE_REQ_URL="https://api.ngc.nvidia.com/v2/resources/$NGC_ORG/$NGC_TEAM/$NGC_RESOURCE/\
   versions/$LATEST_VERSION_ID/files/$NGC_FILENAME" && \
       curl -LO --request GET "${FILE_REQ_URL}" && \
       tar -xf ${NGC_FILENAME} -C ${ISAAC_ROS_WS}/isaac_ros_assets && \
       rm ${NGC_FILENAME}
   fi
   

Build isaac_ros_foundationstereo

Binary PackageBuild from Source

1. Activate the Isaac ROS environment:

   isaac-ros activate
   

2. Install the prebuilt Debian package:

   sudo apt-get update
   

   sudo apt-get install -y ros-jazzy-isaac-ros-foundationstereo && \
      sudo apt-get install -y ros-jazzy-isaac-ros-foundationstereo-models-install
   

3. Download and install the pre-trained FoundationStereo model files:

   sudo apt-get update
   

   ros2 run isaac_ros_foundationstereo_models_install install_foundationstereo_models.sh --eula \
      --model_res high_res
   

   Note

   FoundationStereo supports two fixed resolution configurations:

   • high_res: 576x960 resolution (default)

   • low_res: 320x736 resolution (original training resolution)

   You can set the default model resolution using the FOUNDATIONSTEREO_MODEL_RES environment variable. Use the --model_res argument to override the default or to explicitly select between these options.

   The high_res (576x960) model requires more than 16GB of VRAM, so it only works on GPUs with more than 16GB of VRAM.

Run Launch File

RosbagRealSense CameraZED Camera

1. Continuing inside the Isaac ROS environment, install the following dependencies:

   sudo apt-get update
   

   sudo apt-get install -y ros-jazzy-isaac-ros-examples
   

2. Run the following launch file to spin up a demo using the quickstart rosbag:

   High Resolution (high_res)Low Resolution (low_res)

   ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=foundationstereo \
      engine_file_path:=${ISAAC_ROS_WS:?}/isaac_ros_assets/models/foundationstereo/deployable_v2.0/foundationstereo_576x960.engine
   

3. Open a second terminal and attach to the container:

isaac-ros activate

1. In the second terminal, play the FoundationStereo sample rosbag downloaded in the quickstart assets:

ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationstereo/rosbags/foundationstereo_rosbag \
   --remap /left/camera_info:=/left/camera_info_rect /right/camera_info:=/right/camera_info_rect

Visualize Output

Visualizer ScriptFoxglove

1. Open a terminal and attach to the container:

isaac-ros activate

1. In the terminal, visualize and validate the disparity output using the visualizer script:

ros2 run isaac_ros_foundationstereo isaac_ros_foundationstereo_visualizer.py

The example result is:

Try More Examples

To continue your exploration, check out the following suggested examples:

• Tutorial with Isaac Sim
• Visualize FoundationStereo Disparity Output

Isaac ROS Troubleshooting

For solutions to problems with Isaac ROS, refer to Troubleshooting.

API

Overview

The isaac_ros_foundationstereo package offers functionality to generate a stereo disparity map from stereo images using a trained FoundationStereo model. Given a pair of stereo input images, the package generates a continuous disparity image for the left input image. The package consists of the following node:

FoundationStereoDecoderNode: Processes the model output and generates the disparity map

Usage

ros2 launch isaac_ros_foundationstereo isaac_ros_foundationstereo.launch.py engine_file_path:=<your FoundationStereo engine plan absolute path>

FoundationStereoDecoderNode

ROS Parameters

ROS Parameter	Type	Default	Description
disparity_tensor_name	string	“disparity”	Name of the disparity tensor in the input message.
min_disparity	double	0.0	Minimum disparity value for filtering.
max_disparity	double	10000.0	Maximum disparity value for filtering.

ROS Topics Subscribed

ROS Topic	Interface	Description
tensor_sub	isaac_ros_tensor_list_interfaces/TensorList	Input tensor containing the disparity data.
right/camera_info	NitrosCameraInfo	The right camera model.

ROS Topics Published

ROS Topic	Interface	Description
disparity	NitrosDisparityImage	The processed disparity image.

Input Restrictions

1. The input left and right images must have the same dimension and resolution, and the resolution must be divisible by 32.

Output Interpretations

1. The isaac_ros_foundationstereo package outputs a disparity image with dimension same as the FoundationStereo model output dimension.

   The input images are rescaled and padded to the FoundationStereo model input dimension before inferencing. The disparity output is published as a continuous disparity map.

2. The disparity output is filtered to remove invalid values: - Values below min_disparity (default: 0.0) - Values above max_disparity (default: 10000.0) - Invalid regions (inf, Nan) are set to 0.0

3. The right CameraInfo is used to composite a NitrosDisparityImage. If you only care about the disparity image, and don’t need the baseline and focal length information, you can pass dummy camera messages.