`isaac_ros_foundationpose`

Source code on GitHub.

Quickstart

Set Up Development Environment

Set up your development environment by following the instructions in getting started.

Clone isaac_ros_common under ${ISAAC_ROS_WS}/src.

cd ${ISAAC_ROS_WS}/src && \
   git clone -b release-3.1 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common.git isaac_ros_common

(Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

Warning

We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the Isaac ROS Dev container during installation, which will interrupt the quickstart process.

Download Quickstart Assets

Download quickstart data from NGC:

Make sure required libraries are installed.

sudo apt-get install -y curl jq tar

Then, run these commands to download the asset from NGC:

NGC_ORG="nvidia"
NGC_TEAM="isaac"
PACKAGE_NAME="isaac_ros_foundationpose"
NGC_RESOURCE="isaac_ros_foundationpose_assets"
NGC_FILENAME="quickstart.tar.gz"
MAJOR_VERSION=3
MINOR_VERSION=1
VERSION_REQ_URL="https://catalog.ngc.nvidia.com/api/resources/versions?orgName=$NGC_ORG&teamName=$NGC_TEAM&name=$NGC_RESOURCE&isPublic=true&pageNumber=0&pageSize=100&sortOrder=CREATED_DATE_DESC"
AVAILABLE_VERSIONS=$(curl -s \
    -H "Accept: application/json" "$VERSION_REQ_URL")
LATEST_VERSION_ID=$(echo $AVAILABLE_VERSIONS | jq -r "
    .recipeVersions[]
    | .versionId as \$v
    | \$v | select(test(\"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+$\"))
    | split(\".\") | {major: .[0]|tonumber, minor: .[1]|tonumber, patch: .[2]|tonumber}
    | select(.major == $MAJOR_VERSION and .minor <= $MINOR_VERSION)
    | \$v
    " | sort -V | tail -n 1
)
if [ -z "$LATEST_VERSION_ID" ]; then
    echo "No corresponding version found for Isaac ROS $MAJOR_VERSION.$MINOR_VERSION"
    echo "Found versions:"
    echo $AVAILABLE_VERSIONS | jq -r '.recipeVersions[].versionId'
else
    mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets && \
    FILE_REQ_URL="https://api.ngc.nvidia.com/v2/resources/$NGC_ORG/$NGC_TEAM/$NGC_RESOURCE/\
versions/$LATEST_VERSION_ID/files/$NGC_FILENAME" && \
    curl -LO --request GET "${FILE_REQ_URL}" && \
    tar -xf ${NGC_FILENAME} -C ${ISAAC_ROS_WS}/isaac_ros_assets && \
    rm ${NGC_FILENAME}
fi

Download the pre-trained FoundationPose models from NGC:

wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/isaac/foundationpose/versions/1.0.0/zip -O foundationpose_1.0.0.zip

Copy the models to the required directory:

mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose && \
   unzip foundationpose_1.0.0.zip -d ${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose

Build `isaac_ros_foundationpose`

Launch the Docker container using the run_dev.sh script:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
./scripts/run_dev.sh

Install the prebuilt Debian package:

sudo apt-get install -y ros-humble-isaac-ros-foundationpose

Clone this repository under ${ISAAC_ROS_WS}/src:

cd ${ISAAC_ROS_WS}/src && \
   git clone -b release-3.1 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation.git isaac_ros_pose_estimation

Launch the Docker container using the run_dev.sh script:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh

Use rosdep to install the package’s dependencies:

rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_pose_estimation/isaac_ros_foundationpose --ignore-src -y

Build the package from source:

cd ${ISAAC_ROS_WS}/ && \
   colcon build --symlink-install --packages-up-to isaac_ros_foundationpose

Source the ROS workspace:

Note

Make sure to repeat this step in every terminal created inside the Docker container.

Because this package was built from source, the enclosing workspace must be sourced for ROS to be able to find the package’s contents.
```
source install/setup.bash
```

Run Launch File

Inside the container, convert models from .etlt to TensorRT engine plans:

Convert the refine model:

/opt/nvidia/tao/tao-converter -k foundationpose -t fp16 -e ${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan -p input1,1x160x160x6,1x160x160x6,252x160x160x6 -p input2,1x160x160x6,1x160x160x6,252x160x160x6 -o output1,output2 ${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_model.etlt

Convert the score model:

/opt/nvidia/tao/tao-converter -k foundationpose -t fp16 -e ${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan -p input1,1x160x160x6,1x160x160x6,252x160x160x6 -p input2,1x160x160x6,1x160x160x6,252x160x160x6 -o output1 ${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_model.etlt

Note

The model conversion time varies across different platforms. On Jetson AGX Orin, the engine conversion process takes ~10-15 minutes to complete.

Complete the Isaac ROS RT-DETR tutorial.
Continuing inside the container, install the following dependencies:
```
sudo apt-get install -y ros-humble-isaac-ros-examples
```

Run the following launch files to spin up a demo of this package:

Launch isaac_ros_foundationpose:

ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=foundationpose interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/quickstart_interface_specs.json mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mustard/textured_simple.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mustard/texture_map.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan

Then open another terminal, and enter the Docker container again:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh

Then, play the ROS bag:

ros2 bag play -l  ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/quickstart.bag/

Ensure that you have already set up your RealSense camera using the RealSense setup tutorial. If you have not, please set up the sensor and then restart this quickstart from the beginning.
Complete the Isaac ROS RT-DETR tutorial.
Open a new terminal and launch the Docker container using the run_dev.sh script:
```
cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh
```

Install the following dependencies:

sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-realsense

Place the object in front of the camera and run the launch file:

ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect_depth,foundationpose mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/Mac_and_cheese_0_1.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/materials/textures/baked_mesh_tex0.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan

Ensure that you have already set up your Hawk camera using the Hawk setup tutorial. If you have not, please set up the sensor and then restart this quickstart from the beginning.
Complete the Isaac ROS RT-DETR tutorial.
Open a new terminal and launch the Docker container using the run_dev.sh script:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh

Install the following dependencies:

sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-argus-camera

Place the object in front of the hawk camera and run the launch file:

ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=argus_depth,foundationpose mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/Mac_and_cheese_0_1.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/materials/textures/baked_mesh_tex0.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan

Note

The Hawk pipeline may take more than 10s for pose estimation on some complicated objects.

Visualize Results

Open a new terminal inside the Docker container:
```
cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh
```
1. launch RViz2 to visualize the output
  rviz2 -d $(ros2 pkg prefix isaac_ros_foundationpose --share)/rviz/foundationpose.rviz
2. You should see a RViz2 window open as shown below showing the 3D bounding box overlaid over the input image
1. launch RViz2 to visualize the output
  rviz2 -d $(ros2 pkg prefix isaac_ros_foundationpose --share)/rviz/foundationpose_realsense.rviz
2. You should see a visualization of the 3D pose estimate as shown below:
1. launch RViz2 to visualize the output
  rviz2 -d $(ros2 pkg prefix isaac_ros_foundationpose --share)/rviz/foundationpose_hawk.rviz
2. You should see a visualization of the 3D pose estimate as shown below:

Note

FoundationPose is designed to perform pose estimation on previously unseen objects without model retraining. As a result, inference requires significant GPU compute for first detection. Tracking once an initial pose estimation is determined can be significantly faster, however. Refer to performance of the model for your target platform to determine which model to use.

More powerful discrete GPUs can outperform all other platforms for this task and should be preferred if higher performance is required. Interleaving pose estimation with other tasks rather than running continuously can be a more effective solution as well. Finally, if runtime performance is critical and offline training resources are available, developers can train CenterPose for their own target objects using synthetic data generation and/or real-world data for faster pose estimation.

Try More Examples

To continue your exploration, check out the following suggested examples:

Note

FoundationPose expects the origin frame to be at the center of the mesh.

Troubleshooting

Object Detections are Offset from the Actual Object

Symptom

The object detections are offset from the actual object.

Solution

This may be because the vertices/faces in the .obj file is not at the center of the object. Rather, it could be at the corner/edge of the object. To fix this, follow this tutorial.

Isaac ROS Troubleshooting

For solutions to problems with Isaac ROS, please check here.

Deep Learning Troubleshooting

For solutions to problems with using DNN models, please check here.

API

Usage

ros2 launch isaac_ros_foundationpose isaac_ros_foundationpose.launch.py refine_model_file_path:=<path to refine onnx model> refine_engine_file_path:=<path to refine model .plan> score_model_file_path:=<path to score onnx model> score_engine_file_path:=<path to score model .plan> mesh_file_path:=<path to object mesh file> texture_path:=<path to texture map> launch_rviz:=<enable rviz> launch_bbox_to_mask:=<enable bbox to mask converter> mask_height:=<converted mask height> mask_width:=<converted mask width>

FoundationPose Node

ROS Parameters

ROS Parameter	Type	Default	Description
`mesh_file_path`	`string`	`textured_simple.obj`	The absolute path to the target object mesh file.
`texture_path`	`string`	`textured_map.png`	The absolute Path to the target object texture file.
`min_depth`	`float`	`0.1`	Minimum allowed Z-axis value of pointcloud.
`max_depth`	`float`	`0.4`	Minimum allowed X,Y,Z-axis value of pointcloud to threshold.
`refine_iterations`	`int`	`1`	The number of iterations applied to the refinement model can enhance accuracy. However, more iterations will take longer time for processing.
`refine_model_file_path`	`string`	`/tmp/refine_model.onnx`	The absolute path to the refinement model file in the local file system.
`refine_engine_file_path`	`string`	`/tmp/refine_trt_engine.plan`	The absolute path to either where you want your refinement TensorRT engine plan to be generated or where your pre-generated engine plan file is located.
`score_model_file_path`	`string`	`/tmp/score_model.onnx`	The absolute path to your score model file in the local file system.
`score_engine_file_path`	`string`	`/tmp/score_trt_engine.plan`	The absolute path to either where you want your score TensorRT engine plan to be generated or where your pre-generated engine plan file is located.
`refine_input_tensor_names`	`string`	`['']`	A list of tensor names of refinement model to be bound to specified input bindings names. Bindings occur in sequential order.
`refine_input_binding_names`	`string`	`['']`	A list of input tensor binding names specified by the refinement model.
`score_input_tensor_names`	`string`	`['']`	A list of tensor names of score model to be bound to specified input bindings names. Bindings occur in sequential order.
`score_input_binding_name`	`string`	`['']`	A list of input tensor binding names specified by the score model.
`refine_output_tensor_names`	`string`	`['']`	A list of tensor names to be bound to specified output binding names for the refine model.
`refine_output_binding_names`	`string`	`['']`	A list of output tensor binding names specified by the refine model.
`score_output_tensor_names`	`string`	`['']`	A list of tensor names to be bound to specified output binding names for the score model.
`score_output_binding_names`	`string`	`['']`	A list of output tensor binding names specified by the score model.
`tf_frame_name`	`string`	`fp_object`	Name of the frame that is used when publishing the pose to the TF tree.

ROS Topics Subscribed

ROS Topic	Interface	Description
`pose_estimation/depth_image`	sensor_msgs/Image	The input depth image.
`pose_estimation/segmentation`	sensor_msgs/Image	The input segmentation mask.
`pose_estimation/image`	sensor_msgs/Image	The input color image (rectified).
`pose_estimation/camera_info`	sensor_msgs/CameraInfo	The input image camera_info.

ROS Topics Published

ROS Topic	Interface	Description
`pose_estimation/output`	vision_msgs/Detection3DArray	The output pose estimate.
`pose_estimation/pose_matrix_output`	isaac_ros_tensor_list_interfaces/TensorList	The output pose matrix, used as the input for next frame tracking.

FoundationPose Tracking Node

ROS Parameters

ROS Parameter	Type	Default	Description
`mesh_file_path`	`string`	`textured_simple.obj`	The absolute path to the target object mesh file.
`texture_path`	`string`	`textured_map.png`	The absolute Path to the target object texture file.
`min_depth`	`float`	`0.1`	Minimum allowed Z-axis value of pointcloud.
`max_depth`	`float`	`0.4`	Minimum allowed X,Y,Z-axis value of pointcloud to threshold.
`refine_model_file_path`	`string`	`/tmp/refine_model.onnx`	The absolute path to the refinement model file in the local file system.
`refine_engine_file_path`	`string`	`/tmp/refine_trt_engine.plan`	The absolute path to either where you want your refinement TensorRT engine plan to be generated or where your pre-generated engine plan file is located.
`refine_input_tensor_names`	`string`	`['']`	A list of tensor names of refinement model to be bound to specified input bindings names. Bindings occur in sequential order.
`refine_input_binding_names`	`string`	`['']`	A list of input tensor binding names specified by the refinement model.
`refine_output_tensor_names`	`string`	`['']`	A list of tensor names to be bound to specified output binding names for the refine model.
`refine_output_binding_names`	`string`	`['']`	A list of output tensor binding names specified by the refine model.
`tf_frame_name`	`string`	`fp_object`	Name of the frame that is used when publishing the pose to the TF tree.

ROS Topics Subscribed

ROS Topic	Interface	Description
`tracking/depth_image`	sensor_msgs/Image	The input depth image.
`tracking/pose_input`	isaac_ros_tensor_list_interfaces/TensorList	The input pose estimation matrix from last frame.
`tracking/image`	sensor_msgs/Image	The input color image (rectified).
`tracking/camera_info`	sensor_msgs/CameraInfo	The input image camera_info.

ROS Topics Published

ROS Topic	Interface	Description
`tracking/output`	vision_msgs/Detection3DArray	The output pose estimate.
`tracking/pose_matrix_output`	isaac_ros_tensor_list_interfaces/TensorList	The output pose matrix, used as the input for next frame tracking.

FoundationPose Selector Node

ROS Parameters

ROS Parameter	Type	Default
`reset_period`	`int`	`20000`

ROS Topics Subscribed

ROS Topic	Interface	Description
`depth_image`	sensor_msgs/Image	The input depth image.
`segmentation`	sensor_msgs/Image	The input segmentation mask.
`image`	sensor_msgs/Image	The input color image (rectified).
`camera_info`	sensor_msgs/CameraInfo	The input image camera_info.
`tracking/pose_matrix_output`	isaac_ros_tensor_list_interfaces/TensorList	The input pose matrix comes from last frame tracking.
`pose_estimation/pose_matrix_output`	isaac_ros_tensor_list_interfaces/TensorList	The input pose matrix comes from pose estimation.

ROS Topics Published

ROS Topic	Interface	Description
`pose_estimation/depth_image`	sensor_msgs/Image	The output depth image for pose estimation.
`pose_estimation/segmentation`	sensor_msgs/Image	The output segmentation mask for pose estimation.
`pose_estimation/image`	sensor_msgs/Image	The output color image (rectified) for pose estimation.
`pose_estimation/camera_info`	sensor_msgs/CameraInfo	The output image camera_info for pose estimation.
`tracking/depth_image`	sensor_msgs/Image	The output segmentation mask for tracking.
`tracking/image`	sensor_msgs/Image	The output color image (rectified) for tracking.
`tracking/camera_info`	sensor_msgs/CameraInfo	The output image camera_info for tracking.
`tracking/pose_input`	isaac_ros_tensor_list_interfaces/TensorList	The output pose matrix, used as the input for next frame tracking.

FoundationPose Detection2DToMask Node

ROS Parameters

ROS Parameter	Type	Default	Description
`mask_width`	`int`	`640`	The output mask width. FoundationPose expects the mask to have the same dimensions as the RGB image.
`mask_height`	`int`	`480`	The output mask height. FoundationPose expects the mask to have the same dimensions as the RGB image.

ROS Topics Subscribed

ROS Topic	Interface	Description
`detection2_d`	vision_msgs/Detection2D.msg	The input Detection2D message.
`detection2_d_array`	vision_msgs/Detection2DArray.msg	The input Detection2D Array message. The mask is created using the highest-scoring detection from the input Detection2D Array message.

ROS Topics Published

ROS Topic	Interface	Description
`segmentation`	sensor_msgs/Image	The output binary segmentation mask.

isaac_ros_foundationpose

Quickstart

Set Up Development Environment

Download Quickstart Assets

Build isaac_ros_foundationpose

Run Launch File

Visualize Results

Try More Examples

Troubleshooting

Object Detections are Offset from the Actual Object

Symptom

Solution

Isaac ROS Troubleshooting

Deep Learning Troubleshooting

API

Usage

FoundationPose Node

ROS Parameters

ROS Topics Subscribed

ROS Topics Published

FoundationPose Tracking Node

ROS Parameters

ROS Topics Subscribed

ROS Topics Published

FoundationPose Selector Node

ROS Parameters

ROS Topics Subscribed

ROS Topics Published

FoundationPose Detection2DToMask Node

ROS Parameters

ROS Topics Subscribed

ROS Topics Published

`isaac_ros_foundationpose`

Build `isaac_ros_foundationpose`