Tutorial for FoundationPose with Isaac Sim

Overview

1. Estimate depth using Isaac ROS ESS depth estimation

2. Detect object bounding boxes using Isaac ROS RT-DETR object detection

3. Detect a 3D object using Isaac ROS FoundationPose pose estimation

Note

This tutorial detects the 3D pose of the Mac and Cheese Box mentioned in the SyntheticaDETR Object Classes section. In order to detect other objects from SyntheticaDETR Object Classes, you need to create a mesh and replace the mesh_file_path arguments to this tutorials launch file with the mesh you generated. The tutorial on how to create a mesh using an iPhone can be found here. In order to detect other objects not supported by SyntheticaDETR, you will have to modify the launch file and replace the 2D object detection pipeline with one that supports the object you are trying to detect.

Tutorial Walkthrough

1. Complete the Isaac ROS ESS tutorial.

Note

When going through the Isaac ROS ESS tutorial, ensure you download the light_ess model.

2. Complete the Isaac ROS RT-DETR tutorial.

3. Complete the Isaac ROS FoundationPose Quickstart Guide.

4. Open a new terminal and activate the Isaac ROS environment:

   isaac-ros activate
   

5. Install and launch Isaac Sim following the steps in the Isaac ROS Isaac Sim Setup Guide.

6. Run the launch file.

   ros2 launch isaac_ros_foundationpose isaac_ros_foundationpose_isaac_sim.launch.py
   

7. Press Play to start publishing data from the Isaac Sim.

   

8. Validate that you see a visualization of the 3D pose estimate as shown below: