============== |package_name| ============== :ir_github:` ` Quickstart ---------- Set Up Development Environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. include:: /_snippets/set_up_dev_env.rst Download Quickstart Assets ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Download quickstart data from NGC: :ir_assets:` ` 2. Download the pre-trained :ir_ngc:`FoundationPose ` models from NGC to the required directory: .. code:: bash mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose && \ cd ${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose && \ wget 'https://api.ngc.nvidia.com/v2/models/nvidia/isaac/foundationpose/versions/1.0.0_onnx/files/refine_model.onnx' -O refine_model.onnx && \ wget 'https://api.ngc.nvidia.com/v2/models/nvidia/isaac/foundationpose/versions/1.0.0_onnx/files/score_model.onnx' -O score_model.onnx Build |package_name| ~~~~~~~~~~~~~~~~~~~~ .. tabs:: .. tab:: Binary Package 1. Launch the Docker container using the ``run_dev.sh`` script: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 2. Install the prebuilt Debian package: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-foundationpose .. tab:: Build from Source 1. Clone this repository under ``${ISAAC_ROS_WS}/src``: .. code:: bash cd ${ISAAC_ROS_WS}/src && \ git clone :ir_clone:`` 2. Launch the Docker container using the ``run_dev.sh`` script: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 3. Use ``rosdep`` to install the package's dependencies: :ir_apt: .. code:: bash rosdep update && rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_pose_estimation/isaac_ros_foundationpose --ignore-src -y 4. Build the package from source: .. code:: bash cd ${ISAAC_ROS_WS}/ && \ colcon build --symlink-install --packages-up-to isaac_ros_foundationpose --base-paths ${ISAAC_ROS_WS}/src/isaac_ros_pose_estimation/isaac_ros_foundationpose 5. Source the ROS workspace: .. note:: Make sure to repeat this step in **every** terminal created inside the Docker container. Because this package was built from source, the enclosing workspace must be sourced for ROS to be able to find the package's contents. .. code:: bash source install/setup.bash Run Launch File ~~~~~~~~~~~~~~~ 1. Inside the container, convert models from ``.onnx`` to TensorRT engine plans: Convert the refine model: .. code:: bash /usr/src/tensorrt/bin/trtexec --onnx=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_model.onnx --saveEngine=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan --minShapes=input1:1x160x160x6,input2:1x160x160x6 --optShapes=input1:1x160x160x6,input2:1x160x160x6 --maxShapes=input1:42x160x160x6,input2:42x160x160x6 Convert the score model: .. code:: bash /usr/src/tensorrt/bin/trtexec --onnx=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_model.onnx --saveEngine=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan --minShapes=input1:1x160x160x6,input2:1x160x160x6 --optShapes=input1:1x160x160x6,input2:1x160x160x6 --maxShapes=input1:252x160x160x6,input2:252x160x160x6 .. note:: FoundationPose TensorRT engines are running with FP32 precision in TensorRT 10.3+ versions due to FP16 precision loss. .. note:: At least **7.5 GB** of free GPU memory space is required for model conversion. The model conversion time varies across different platforms. On Jetson AGX Orin, the engine conversion process takes ~10-15 minutes to complete. .. tabs:: .. group-tab:: Rosbag 2. Complete the :ref:`Isaac ROS RT-DETR tutorial `. 3. Continuing inside the container, install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples 4. Run the following launch files to spin up a demo of this package: Launch ``isaac_ros_foundationpose``: .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=foundationpose interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/quickstart_interface_specs.json mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mustard/textured_simple.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mustard/texture_map.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan Then open **another** terminal, and enter the Docker container again: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh Then, play the rosbag: .. code:: bash ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/quickstart.bag/ .. group-tab:: RealSense Camera 2. Ensure that you have already set up your RealSense camera using the :doc:`RealSense setup tutorial `. If you have not, please set up the sensor and then restart this quickstart from the beginning. 3. Complete the :ref:`Isaac ROS RT-DETR tutorial `. 4. Open a new terminal and launch the Docker container using the ``run_dev.sh`` script: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 5. Install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-realsense 6. Place the object in front of the camera and run the launch file: .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect_depth,foundationpose mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/Mac_and_cheese_0_1.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/materials/textures/baked_mesh_tex0.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan .. group-tab:: Hawk Camera 2. Ensure that you have already set up your Hawk camera using the :doc:`Hawk setup tutorial `. If you have not, please set up the sensor and then restart this quickstart from the beginning. 3. Complete the :ref:`Isaac ROS RT-DETR tutorial `. 4. Open a new terminal and launch the Docker container using the ``run_dev.sh`` script: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 5. Install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-argus-camera 6. Place the object in front of the hawk camera and run the launch file: .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=argus_depth,foundationpose mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/Mac_and_cheese_0_1.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/materials/textures/baked_mesh_tex0.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan .. note:: The Hawk pipeline may take more than 10s for pose estimation on some complicated objects. Visualize Results ~~~~~~~~~~~~~~~~~ 1. Open a **new** terminal inside the Docker container: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh .. tabs:: .. group-tab:: Rosbag 2. launch ``RViz2`` to visualize the output .. code:: bash rviz2 -d $(ros2 pkg prefix isaac_ros_foundationpose --share)/rviz/foundationpose.rviz 3. You should see a ``RViz2`` window open as shown below showing the 3D bounding box overlaid over the input image .. figure:: :ir_lfs:`` :width: 600px :align: center .. group-tab:: RealSense Camera 2. launch ``RViz2`` to visualize the output .. code:: bash rviz2 -d $(ros2 pkg prefix isaac_ros_foundationpose --share)/rviz/foundationpose_realsense.rviz 3. You should see a visualization of the 3D pose estimate as shown below: .. figure:: :ir_lfs:`` :alt: Isaac ROS FoundationPose RealSense Pose Estimation :align: center .. group-tab:: Hawk Camera 2. launch ``RViz2`` to visualize the output .. code:: bash rviz2 -d $(ros2 pkg prefix isaac_ros_foundationpose --share)/rviz/foundationpose_hawk.rviz 3. You should see a visualization of the 3D pose estimate as shown below: .. figure:: :ir_lfs:`` :alt: Isaac ROS FoundationPose Hawk Pose Estimation :align: center .. note:: FoundationPose is designed to perform pose estimation on previously unseen objects without model retraining. As a result, inference requires significant GPU compute for first detection. Tracking once an initial pose estimation is determined can be significantly faster, however. Refer to `performance `__ of the model for your target platform to determine which model to use. More powerful discrete GPUs can outperform all other platforms for this task and should be preferred if higher performance is required. Interleaving pose estimation with other tasks rather than running continuously can be a more effective solution as well. Finally, if runtime performance is critical and offline training resources are available, developers can train :doc:`CenterPose ` for their own target objects using synthetic data generation and/or real-world data for faster pose estimation. Try More Examples ----------------- To continue your exploration, check out the following suggested examples: .. toctree:: :maxdepth: 1 Tutorial with Isaac Sim Tutorial to launch FoundationPose Tracking Tutorial to create your own 3D object mesh in .obj .. note:: ``FoundationPose`` expects the origin frame to be at the center of the mesh. .. toctree:: :maxdepth: 1 Tutorial to shift the Mesh origin frame to the center of the Mesh using Meshlab Tutorial to convert a 3D mesh from .usd to .obj using Isaac Sim Tutorial to simplify a 3D mesh using Meshlab Troubleshooting --------------- .. include:: /repositories_and_packages/isaac_ros_pose_estimation/_snippets/troubleshooting.rst Isaac ROS Troubleshooting ~~~~~~~~~~~~~~~~~~~~~~~~~ For solutions to problems with Isaac ROS, please check :doc:`here `. Deep Learning Troubleshooting ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For solutions to problems with using DNN models, please check :doc:`here `. API ---- Usage ~~~~~ .. code:: bash ros2 launch isaac_ros_foundationpose isaac_ros_foundationpose.launch.py refine_engine_file_path:= score_engine_file_path:= mesh_file_path:= texture_path:= launch_rviz:= launch_bbox_to_mask:= mask_height:= mask_width:= FoundationPose Node ~~~~~~~~~~~~~~~~~~~ ROS Parameters ^^^^^^^^^^^^^^ =============================== ============== ========================================================================================================== =================================================================================================================================================================================== ROS Parameter Type Default Description =============================== ============== ========================================================================================================== =================================================================================================================================================================================== ``mesh_file_path`` ``string`` ``textured_simple.obj`` The absolute path to the target object mesh file. ``texture_path`` ``string`` ``textured_map.png`` The absolute Path to the target object texture file. ``min_depth`` ``float`` ``0.1`` Minimum allowed Z-axis value of pointcloud. ``max_depth`` ``float`` ``0.4`` Minimum allowed X,Y,Z-axis value of pointcloud to threshold. ``refine_iterations`` ``int`` ``1`` The number of iterations applied to the refinement model can enhance accuracy. However, more iterations will take longer time for processing. ``refine_model_file_path`` ``string`` ``/tmp/refine_model.onnx`` The absolute path to the refinement model file in the local file system. ``refine_engine_file_path`` ``string`` ``/tmp/refine_trt_engine.plan`` The absolute path to either where you want your refinement TensorRT engine plan to be generated or where your pre-generated engine plan file is located. ``score_model_file_path`` ``string`` ``/tmp/score_model.onnx`` The absolute path to your score model file in the local file system. ``score_engine_file_path`` ``string`` ``/tmp/score_trt_engine.plan`` The absolute path to either where you want your score TensorRT engine plan to be generated or where your pre-generated engine plan file is located. ``refine_input_tensor_names`` ``string`` ``['']`` A list of tensor names of refinement model to be bound to specified input bindings names. Bindings occur in sequential order. ``refine_input_binding_names`` ``string`` ``['']`` A list of input tensor binding names specified by the refinement model. ``score_input_tensor_names`` ``string`` ``['']`` A list of tensor names of score model to be bound to specified input bindings names. Bindings occur in sequential order. ``score_input_binding_name`` ``string`` ``['']`` A list of input tensor binding names specified by the score model. ``refine_output_tensor_names`` ``string`` ``['']`` A list of tensor names to be bound to specified output binding names for the refine model. ``refine_output_binding_names`` ``string`` ``['']`` A list of output tensor binding names specified by the refine model. ``score_output_tensor_names`` ``string`` ``['']`` A list of tensor names to be bound to specified output binding names for the score model. ``score_output_binding_names`` ``string`` ``['']`` A list of output tensor binding names specified by the score model. ``tf_frame_name`` ``string`` ``fp_object`` Name of the frame that is used when publishing the pose to the TF tree. =============================== ============== ========================================================================================================== =================================================================================================================================================================================== ROS Topics Subscribed ^^^^^^^^^^^^^^^^^^^^^ ================================ ==================================================================================================================================================================== ==================================================================================== ROS Topic Interface Description ================================ ==================================================================================================================================================================== ==================================================================================== ``pose_estimation/depth_image`` `sensor_msgs/Image `__ The input depth image. ``pose_estimation/segmentation`` `sensor_msgs/Image `__ The input segmentation mask. ``pose_estimation/image`` `sensor_msgs/Image `__ The input color image (rectified). ``pose_estimation/camera_info`` `sensor_msgs/CameraInfo `__ The input image camera_info. ================================ ==================================================================================================================================================================== ==================================================================================== ROS Topics Published ^^^^^^^^^^^^^^^^^^^^ ====================================== ================================================================================================================================ ======================================================================================================= ROS Topic Interface Description ====================================== ================================================================================================================================ ======================================================================================================= ``pose_estimation/output`` `vision_msgs/Detection3DArray `__ The output pose estimate. ``pose_estimation/pose_matrix_output`` :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList ` The output pose matrix, used as the input for next frame tracking. ====================================== ================================================================================================================================ ======================================================================================================= FoundationPose Tracking Node ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ROS Parameters ^^^^^^^^^^^^^^ =============================== ============== ========================================================================================================= ==================================================================================================================================================================================== ROS Parameter Type Default Description =============================== ============== ========================================================================================================= ==================================================================================================================================================================================== ``mesh_file_path`` ``string`` ``textured_simple.obj`` The absolute path to the target object mesh file. ``texture_path`` ``string`` ``textured_map.png`` The absolute Path to the target object texture file. ``min_depth`` ``float`` ``0.1`` Minimum allowed Z-axis value of pointcloud. ``max_depth`` ``float`` ``0.4`` Minimum allowed X,Y,Z-axis value of pointcloud to threshold. ``refine_model_file_path`` ``string`` ``/tmp/refine_model.onnx`` The absolute path to the refinement model file in the local file system. ``refine_engine_file_path`` ``string`` ``/tmp/refine_trt_engine.plan`` The absolute path to either where you want your refinement TensorRT engine plan to be generated or where your pre-generated engine plan file is located. ``refine_input_tensor_names`` ``string`` ``['']`` A list of tensor names of refinement model to be bound to specified input bindings names. Bindings occur in sequential order. ``refine_input_binding_names`` ``string`` ``['']`` A list of input tensor binding names specified by the refinement model. ``refine_output_tensor_names`` ``string`` ``['']`` A list of tensor names to be bound to specified output binding names for the refine model. ``refine_output_binding_names`` ``string`` ``['']`` A list of output tensor binding names specified by the refine model. ``tf_frame_name`` ``string`` ``fp_object`` Name of the frame that is used when publishing the pose to the TF tree. =============================== ============== ========================================================================================================= ==================================================================================================================================================================================== ROS Topics Subscribed ^^^^^^^^^^^^^^^^^^^^^ =============================== ==================================================================================================================================================================== ==================================================================================== ROS Topic Interface Description =============================== ==================================================================================================================================================================== ==================================================================================== ``tracking/depth_image`` `sensor_msgs/Image `__ The input depth image. ``tracking/pose_input`` :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList ` The input pose estimation matrix from last frame. ``tracking/image`` `sensor_msgs/Image `__ The input color image (rectified). ``tracking/camera_info`` `sensor_msgs/CameraInfo `__ The input image camera_info. =============================== ==================================================================================================================================================================== ==================================================================================== ROS Topics Published ^^^^^^^^^^^^^^^^^^^^ ================================ ================================================================================================================================ ======================================================================================================= ROS Topic Interface Description ================================ ================================================================================================================================ ======================================================================================================= ``tracking/output`` `vision_msgs/Detection3DArray `__ The output pose estimate. ``tracking/pose_matrix_output`` :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList ` The output pose matrix, used as the input for next frame tracking. ================================ ================================================================================================================================ ======================================================================================================= FoundationPose Selector Node ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ROS Parameters ^^^^^^^^^^^^^^ ============================== ============== ======================================================================================================== ROS Parameter Type Default ============================== ============== ======================================================================================================== ``reset_period`` ``int`` ``20000`` ============================== ============== ======================================================================================================== ROS Topics Subscribed ^^^^^^^^^^^^^^^^^^^^^ ====================================== ==================================================================================================================================================================== ==================================================================================== ROS Topic Interface Description ====================================== ==================================================================================================================================================================== ==================================================================================== ``depth_image`` `sensor_msgs/Image `__ The input depth image. ``segmentation`` `sensor_msgs/Image `__ The input segmentation mask. ``image`` `sensor_msgs/Image `__ The input color image (rectified). ``camera_info`` `sensor_msgs/CameraInfo `__ The input image camera_info. ``tracking/pose_matrix_output`` :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList ` The input pose matrix comes from last frame tracking. ``pose_estimation/pose_matrix_output`` :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList ` The input pose matrix comes from pose estimation. ====================================== ==================================================================================================================================================================== ==================================================================================== ROS Topics Published ^^^^^^^^^^^^^^^^^^^^ ================================ =============================================================================================================================== ======================================================================================================= ROS Topic Interface Description ================================ =============================================================================================================================== ======================================================================================================= ``pose_estimation/depth_image`` `sensor_msgs/Image `__ The output depth image for pose estimation. ``pose_estimation/segmentation`` `sensor_msgs/Image `__ The output segmentation mask for pose estimation. ``pose_estimation/image`` `sensor_msgs/Image `__ The output color image (rectified) for pose estimation. ``pose_estimation/camera_info`` `sensor_msgs/CameraInfo `__ The output image camera_info for pose estimation. ``tracking/depth_image`` `sensor_msgs/Image `__ The output segmentation mask for tracking. ``tracking/image`` `sensor_msgs/Image `__ The output color image (rectified) for tracking. ``tracking/camera_info`` `sensor_msgs/CameraInfo `__ The output image camera_info for tracking. ``tracking/pose_input`` :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList ` The output pose matrix, used as the input for next frame tracking. ================================ =============================================================================================================================== ======================================================================================================= FoundationPose Detection2DToMask Node ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ROS Parameters ^^^^^^^^^^^^^^ ============================== =============== ========================================================================================================= =================================================================================================================================================================================== ROS Parameter Type Default Description ============================== =============== ========================================================================================================= =================================================================================================================================================================================== ``mask_width`` ``int`` ``640`` The output mask width. FoundationPose expects the mask to have the same dimensions as the RGB image. ``mask_height`` ``int`` ``480`` The output mask height. FoundationPose expects the mask to have the same dimensions as the RGB image. ============================== =============== ========================================================================================================= =================================================================================================================================================================================== ROS Topics Subscribed ^^^^^^^^^^^^^^^^^^^^^ =============================== ==================================================================================================================================================================== ====================================================================================================== ROS Topic Interface Description =============================== ==================================================================================================================================================================== ====================================================================================================== ``detection2_d`` `vision_msgs/Detection2D.msg `__ The input Detection2D message. ``detection2_d_array`` `vision_msgs/Detection2DArray.msg `__ The input Detection2D Array message. The mask is created using the highest-scoring detection from the input Detection2D Array message. =============================== ==================================================================================================================================================================== ====================================================================================================== ROS Topics Published ^^^^^^^^^^^^^^^^^^^^ ================================ =============================================================================================================================== ======================================================================================================= ROS Topic Interface Description ================================ =============================================================================================================================== ======================================================================================================= ``segmentation`` `sensor_msgs/Image `__ The output binary segmentation mask. ================================ =============================================================================================================================== ======================================================================================================= .. |package_name| replace:: ``isaac_ros_foundationpose``