Tutorial for Segment Anything with Isaac Sim

Overview

This tutorial demonstrates how to:

1. Setup and stream images using Isaac Sim.

2. Segment objects using Isaac ROS Segment Anything.

3. Detect object bounding boxes using Isaac ROS YoloV8 object detection.

Tutorial Walkthrough

1. Complete the quickstart up until model preparation step.

2. Complete the Isaac ROS YoloV8 tutorial up until the build step.

3. Launch the Docker container using the run_dev.py script:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
     ./scripts/run_dev.py
   

4. Install and launch Isaac Sim following the steps in the Isaac ROS Isaac Sim Setup Guide.

5. Press Play to start publishing data from the Isaac Sim.

   

6. Run the following launch files to start the inferencing:

   ros2 launch isaac_ros_segment_anything isaac_ros_segment_anything_isaac_sim.launch.py model_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/yolov8/yolov8s.onnx engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/yolov8/yolov8s.plan confidence_threshold:=0.25 nms_threshold:=0.45 model_repository_paths:=[${ISAAC_ROS_WS}/isaac_ros_assets/models]
   

7. Start a new terminal and attach to the container.

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
    ./scripts/run_dev.py
   

   Then run the Python script to generate the colored segmentation mask from raw mask.

   ros2 run isaac_ros_segment_anything visualize_mask.py
   

8. Visualize and validate the output of the package by launching rqt_image_view. In another terminal enter the Docker container:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
    ./scripts/run_dev.py

Then launch rqt_image_view:

ros2 run rqt_image_view rqt_image_view

Inside the rqt_image_view GUI, change the topic to /segment_anything/colored_segmentation_mask to view a colorized segmentation mask.