Tutorial for DNN Image Segmentation with NITROS Bridge

Overview

This tutorial walks you through how to run the isaac_ros_unet on ROS 2 Humble, while playing rosbag and getting the results from ROS Noetic through isaac_ros_nitros_bridge. The tutorial is based on the ROS1 Bridge documentation.

Note

ROS actions are not supported by the ROS1 Bridge

Steps

Note

The ROS Noetic environment requires more storage space than the usual Isaac ROS Dev Docker-based development environment. We recommend at least 60 GB, to account for the size of the ROS Noetic container and datasets.

On Jetson platforms, NVMe SSD storage is required for sufficient and fast storage. See here

  1. Set up your development environment by following the instructions here.

  2. Clone the isaac_ros_nitros_bridge repository and its dependencies under Isaac ROS workspace:

    mkdir -p $ISAAC_ROS_WS/src && cd $ISAAC_ROS_WS/src && \
git clone https://github.com/ros2/ros1_bridge && \
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common.git && \
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros.git && \
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros_bridge.git

  3. Create a ROS 1 workspace for experimenting with Isaac ROS:

    mkdir -p  ${HOME}/workspaces/isaac_ros_1-dev/src
echo "export ISAAC_ROS_1_WS=${HOME}/workspaces/isaac_ros_1-dev/" >> ~/.bashrc
source ~/.bashrc

    Note

    Note that we are going to use the ISAAC_ROS_1_WS environmental variable in the future to refer to this ROS 1 workspace directory.

  4. Clone the isaac_ros_noetic_interfaces and isaac_ros_nitros_bridge repositories into the isaac_ros_1-dev directory:

    cd $ISAAC_ROS_1_WS/src && \
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_noetic_interfaces.git && \
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros_bridge.git

  5. Configure the container created by isaac_ros_common/scripts/run_dev.sh to include ros1_noetic. Create the .isaac_ros_common-config file in the isaac_ros_common/scripts directory:

    cd ${ISAAC_ROS_WS}/src/isaac_ros_common/scripts && \
touch .isaac_ros_common-config && \
echo CONFIG_IMAGE_KEY=ros2_humble.ros1_noetic >> .isaac_ros_common-config && \
echo "CONFIG_DOCKER_SEARCH_DIRS=(../../isaac_ros_nitros_bridge/docker)" >> .isaac_ros_common-config

  6. Remove the following two lines from workspace-entrypoint.sh in the isaac_ros_common/docker/scripts directory because ros1_bridge requires a specific setup.bash script sourcing order:

    echo "source /opt/ros/${ROS_DISTRO}/setup.bash" >> ~/.bashrc
source /opt/ros/${ROS_DISTRO}/setup.bash

  7. Launch the Docker container:

    cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
./scripts/run_dev.sh ${HOME}/workspaces

  8. Build the custom ROS 1 packages and the ROS 2 packages. But do NOT build the ros1_bridge package:

    source /opt/ros/noetic/setup.bash && \
cd /workspaces/isaac_ros-dev/isaac_ros_1-dev && \
catkin_make_isolated --install --ignore-pkg isaac_ros_nitros_bridge_ros2 && \
source /opt/ros/humble/setup.bash && \
cd /workspaces/isaac_ros-dev/isaac_ros-dev && \
colcon build --symlink-install --packages-skip ros1_bridge

    Note

    You will see the following warning messages while building under ROS 2 Humble. This is expected as isaac_ros_nitros_bridge_ros1 and isaac_ros_ros1_forward are ROS Noetic packages and they are skipped from the building. Please ignore those warnings.

WARNING:colcon.colcon_cmake.task.cmake.build:Could not run installation step for package 'isaac_ros_nitros_bridge_ros1' because it has no 'install' target
WARNING:colcon.colcon_cmake.task.cmake.build:Could not run installation step for package 'isaac_ros_ros1_forward' because it has no 'install' target

  1. Attach a second terminal to the container:

    cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
./scripts/run_dev.sh ${HOME}/workspaces

    Note

    Do NOT close the first terminal when you attach this second terminal to the container.

  2. In the second terminal, build ros1_bridge and source the workspace:

    source isaac_ros_1-dev/install_isolated/setup.bash && \
source isaac_ros-dev/install/setup.bash && \
cd /workspaces/isaac_ros-dev/isaac_ros-dev && \
colcon build --symlink-install --packages-select ros1_bridge --cmake-force-configure && \
source install/setup.bash

Now, we can setup isaac_ros_unet and run it with isaac_ros_nitros_bridge.

  1. Clone isaac_ros_unet and its dependencies under /workspaces/isaac_ros-dev/isaac_ros-dev/src:

    cd /workspaces/isaac_ros-dev/isaac_ros-dev/src && \
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation.git && \
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_inference.git && \
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_pipeline.git

  2. Complete isaac_ros_unet Quickstart step 5-6 to download image segmentation models and convert it to TensorRT engine.

  3. Create a file called /tmp/models/peoplesemsegnet_shuffleseg/config.pbtxt by copying the sample Triton config file:

    cp /workspaces/isaac_ros-dev/isaac_ros-dev/src/isaac_ros_image_segmentation/resources/peoplesemsegnet_shuffleseg_config.pbtxt /tmp/models/peoplesemsegnet_shuffleseg/config.pbtxt

  4. Inside the container, build isaac_ros_unet and source the workspace:

    cd /workspaces/isaac_ros-dev/isaac_ros-dev && \
  colcon build --symlink-install --packages-up-to isaac_ros_unet && \
  source install/setup.bash

  5. Run the following launch files to spin up a demo of this package:

    ros2 launch isaac_ros_unet isaac_ros_unet_triton.launch.py model_name:=peoplesemsegnet_shuffleseg model_repository_paths:=['/tmp/models'] input_binding_names:=['input_2:0'] output_binding_names:=['argmax_1'] network_output_type:='argmax' input_image_width:=1200 input_image_height:=632

  6. Attach a fourth terminal to the Docker container:

    cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
./scripts/run_dev.sh ${HOME}/workspaces

  7. Launch the image converter launch file of isaac_ros_nitros_bridge:

    cd /workspaces/isaac_ros-dev/isaac_ros-dev && \
  source install/setup.bash && \
  ros2 launch isaac_ros_nitros_bridge_ros2 isaac_ros_image_converter.launch.py ros1_ws_path:=/workspaces/isaac_ros-dev/isaac_ros_1-dev/ pub_image_name:=image sub_image_name:=unet/colored_segmentation_mask

  8. Attach a fourth terminal to the Docker container:

    cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
./scripts/run_dev.sh ${HOME}/workspaces

  9. Pull down a rosbag of image segmentation sample:

    cd /workspaces/isaac_ros-dev/isaac_ros_1-dev/src/isaac_ros_nitros_bridge && \
  git lfs pull -X "" -I "resources/unet_sample_data_ros1.bag"

  10. Play the rosbag under ROS Noetic:

    cd /workspaces/isaac_ros-dev/isaac_ros_1-dev/ && \
  source install_isolated/setup.bash && \
  rosbag play -l "src/isaac_ros_nitros_bridge/resources/unet_sample_data_ros1.bag"

  11. Attach a fifth terminal to the Docker container:

    cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
./scripts/run_dev.sh ${HOME}/workspaces

  12. Source ros_noetic, download rviz for ROS Noetic:

    source /opt/ros/noetic/setup.bash && \
  sudo apt-get update && sudo apt-get install ros-noetic-rviz

  13. Run rviz under ROS Noetic, add the colorized segmentation topic /ros1_output_image sent from ROS 2 humble and visualize:

    rviz

  14. You should be able to see the colorized segmentation:

    https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/peoplesemsegnet_shuffleseg_rqt.png/