isaac_ros_segment_anything2

Source code available on GitHub.

Quickstart

Set Up Development Environment

1. Set up your development environment by following the instructions in getting started.

2. (Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

   Note

   We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the development environment during installation, which will interrupt the quickstart process.

Download Quickstart Assets

1. Download quickstart data from NGC:

   Make sure required libraries are installed.

   sudo apt-get install -y curl jq tar
   

   Then, run these commands to download the asset from NGC:

   NGC_ORG="nvidia"
   NGC_TEAM="isaac"
   PACKAGE_NAME="isaac_ros_segment_anything2"
   NGC_RESOURCE="isaac_ros_segment_anything2_assets"
   NGC_FILENAME="quickstart.tar.gz"
   MAJOR_VERSION=4
   MINOR_VERSION=0
   VERSION_REQ_URL="https://catalog.ngc.nvidia.com/api/resources/versions?orgName=$NGC_ORG&teamName=$NGC_TEAM&name=$NGC_RESOURCE&isPublic=true&pageNumber=0&pageSize=100&sortOrder=CREATED_DATE_DESC"
   AVAILABLE_VERSIONS=$(curl -s \
       -H "Accept: application/json" "$VERSION_REQ_URL")
   LATEST_VERSION_ID=$(echo $AVAILABLE_VERSIONS | jq -r "
       .recipeVersions[]
       | .versionId as \$v
       | \$v | select(test(\"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+$\"))
       | split(\".\") | {major: .[0]|tonumber, minor: .[1]|tonumber, patch: .[2]|tonumber}
       | select(.major == $MAJOR_VERSION and .minor <= $MINOR_VERSION)
       | \$v
       " | sort -V | tail -n 1
   )
   if [ -z "$LATEST_VERSION_ID" ]; then
       echo "No corresponding version found for Isaac ROS $MAJOR_VERSION.$MINOR_VERSION"
       echo "Found versions:"
       echo $AVAILABLE_VERSIONS | jq -r '.recipeVersions[].versionId'
   else
       mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets && \
       FILE_REQ_URL="https://api.ngc.nvidia.com/v2/resources/$NGC_ORG/$NGC_TEAM/$NGC_RESOURCE/\
   versions/$LATEST_VERSION_ID/files/$NGC_FILENAME" && \
       curl -LO --request GET "${FILE_REQ_URL}" && \
       tar -xf ${NGC_FILENAME} -C ${ISAAC_ROS_WS}/isaac_ros_assets && \
       rm ${NGC_FILENAME}
   fi
   

Build isaac_ros_segment_anything2

Binary PackageBuild from Source

1. Activate the Isaac ROS environment:

   isaac-ros activate
   

2. Install the prebuilt Debian package:

   sudo apt-get update
   

   sudo apt-get install -y ros-jazzy-isaac-ros-segment-anything2
   

Prepare Segment Anything2 ONNX Model

1. Make a directory to place models (inside the Isaac ROS environment):

   mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets/models/segment_anything2/1/
   

2. Generate the ONNX file for SAM2:

Note

The official video predictor pipeline for SAM2 is not ONNX exportable. To address this limitation, we have developed a custom wrapper for SAM2 that replicates the behavior of the official video predictor pipeline using ONNX-exportable operations. The key difference is that our implementation uses memories only from the conditioned frame and the last frame where the object was detected. This wrapper is ONNX exportable and can be found here: sam2_wrapper.py

Note

SAM2 is only supported with Triton using the ONNX backend. The following step is to be performed only on x86_64. You must copy the generated ONNX file to a Jetson, in case you want to run the package on the device.

To create ONNX file for SAM2. Download the PyTorch weights from official SAM2 repo. Then, copy the downloaded model weights into the container. The model is expected to be downloaded to ~/Downloads outside the Isaac ROS environment.

This example uses sam2.1_hiera_tiny.pt, which must be copied into ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_segment_anything2 inside the Isaac ROS environment:

On x86_64:

# Outside container
mv ~/Downloads/sam2.1_hiera_tiny.pt ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_segment_anything2/sam2.1_hiera_tiny.pt

# Inside container, install sam2
cd ${ISAAC_ROS_WS} && pip install --break-system-packages git+https://github.com/facebookresearch/sam2.git -v

Note

Run below command if you want to convert the model to ONNX using fp16 precision. The fp16 precision can be used to reduce the memory footprint of the model and improve the inference speed with some degradation in accuracy.

cd ${ISAAC_ROS_WS} && pip install --break-system-packages onnxconverter-common==1.14.0

The below command will convert the model to ONNX using fp16 precision. Remove –fp16 flag at the end if you want to convert the model to ONNX using fp32 precision.

# Inside container
ros2 run isaac_ros_segment_anything2 sam2_onnx_exporter.py \
--checkpoint ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_segment_anything2/sam2.1_hiera_tiny.pt \
--output ${ISAAC_ROS_WS}/isaac_ros_assets/models/segment_anything2/1/model.onnx --fp16

1. Copy the config file:

   cp ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_segment_anything2/sam2_config.pbtxt ${ISAAC_ROS_WS}/isaac_ros_assets/models/segment_anything2/config.pbtxt
   

Run Launch File

RosbagRealSense CameraZED Camera

1. Continuing inside the Isaac ROS environment, install the following dependencies:

   sudo apt-get update
   

   sudo apt-get install -y ros-jazzy-isaac-ros-examples
   

2. Run the following launch file to spin up a demo of this package:

   cd ${ISAAC_ROS_WS} && \
      ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=segment_anything2 interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_segment_anything2/quickstart_interface_specs.json sam2_model_repository_paths:=[${ISAAC_ROS_WS}/isaac_ros_assets/models]
   

3. Open a second terminal and activate the Isaac ROS environment:

   isaac-ros activate
   

4. Add the object for tracking using the utility script:

   cd ${ISAAC_ROS_WS} && \
   ros2 run isaac_ros_segment_anything2 add_object.py --object-id "bottle" bbox --x-center 140.5 --y-center 306 --width 83 --height 86
   

5. Open another terminal and activate the Isaac ROS environment:

   isaac-ros activate
   

6. Play the rosbag:

   ros2 bag play ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_segment_anything2/quickstart.bag/
   

Visualize Results

1. Open a new terminal inside the Isaac ROS environment:

   isaac-ros activate
   

2. Run the Python script to generate the colored segmentation mask from raw mask.

   ros2 run isaac_ros_segment_anything visualize_mask.py  --remap /segment_anything/raw_segmentation_mask:=/segment_anything2/raw_segmentation_mask /yolov8_encoder/resize/image:=/image_rect
   

3. Open another terminal and activate the Isaac ROS environment:

   isaac-ros activate
   

4. Launch rqt_image_view:

   ros2 run rqt_image_view rqt_image_view /segment_anything/colored_segmentation_mask
   

   

   Note

   The raw segmentation mask is also published to /segment_anything2/raw_segmentation_mask. However, the raw pixels correspond to the class labels and so the output is unsuitable for human visual inspection.

Troubleshooting

Isaac ROS Troubleshooting

For solutions to problems with Isaac ROS, see here.

Deep Learning Troubleshooting

For solutions to problems with using DNN models, see here.

API

Usage

A single launch file is provided for this package. The launch file launches isaac_ros_triton with ONNX backend. Only the ONNX backend with Triton is supported for this model.

Note

For your specific application, these launch files may need to be modified. Please consult the available components to see the configurable parameters.

Launch File	Components Used
isaac_ros_segment_anything2_core.launch.py	ResizeNode, PadNode, ImageFormatConverterNode, ImageToTensorNode, ImageTensorNormalizeNode, InterleavedToPlanarNode, ReshapeNode, TritonNode, SegmentAnythingDecoderNode, SegmentAnything2DataEncoderNode

SegmentAnything2DataEncoderNode

ROS Parameters

ROS Parameter	Type	Default	Description
max_num_objects	int32_t	10	Max Number of Objects that can be tracked.
orig_img_dims	double list	[480,640]	Dimensions of the Original Image in [H,W] format. Please note, expectation is each bbox/point is in the coordinate system of given dimension image.

ROS Topics Subscribed

ROS Topic	Interface	Description
/memory	isaac_ros_tensor_list_interfaces/TensorList	Memory data from the SAM2 model that helps track objects between video frames.
/image	isaac_ros_tensor_list_interfaces/TensorList	Input image tensor.

ROS Topics Published

ROS Topic	Interface	Description
/encoded_data	isaac_ros_tensor_list_interfaces/TensorList	Tensor List which has all the required tensors for SAM2 inference.

ROS Services Advertised

ROS Service	Interface	Description
add_objects	isaac_ros_segment_anything2_interfaces/AddObjects	A service to add objects to the segmentation model.
remove_object	isaac_ros_segment_anything2_interfaces/RemoveObject	A service to remove objects from the segmentation model.