Install Packages and Perception Models#

First complete the environment setup. These instructions reuse the manipulation workflow’s source-build instructions. Build the Isaac ROS packages and their source dependencies in the same workspace.

Select the middleware for your target before building. Use RMW_IMPLEMENTATION=rmw_fastrtps_cpp for Isaac Sim, or follow the physical-robot middleware instructions for real-robot deployment.

Install Manipulation Packages#

Build the manipulation packages and their Isaac ROS dependencies from source. Place the dependency repositories in ${ISAAC_ROS_WS}/src before running rosdep so --ignore-src resolves those packages from the workspace. The dependency installation commands below provide system and ROS prerequisites.

  1. Clone the isaac_ros_manipulation repository under ${ISAAC_ROS_WS}/src:

    cd ${ISAAC_ROS_WS}/src && git clone --recursive -b release-r1 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_manipulation.git isaac_ros_manipulation

  2. Activate the Isaac ROS environment:

    isaac-ros activate

  3. Install system dependencies:

    sudo apt update && sudo apt install -y python3-pip python3-git && pip install --break-system-packages pyvers==0.2.2

  4. Use rosdep to install the package’s dependencies:

    sudo apt-get update
rosdep update && rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_manipulation/isaac_ros_manipulation_bringup --ignore-src -y

  5. Build the package from source, with automatic asset installation disabled. Prepare any perception models required by your workflow separately.

    cd ${ISAAC_ROS_WS}
ISAAC_ROS_SKIP_ASSET_INSTALL=1 MANIPULATOR_INSTALL_ASSETS=0 colcon build --symlink-install --packages-up-to isaac_ros_manipulation_bringup

  6. Source the ROS workspace and set your ROS_DOMAIN_ID:

    Note

    Repeat this step in every terminal that you open inside the Isaac ROS environment.

    Because you built this package from source, you must source the enclosing workspace so that ROS can find the package’s contents.

    source install/setup.bash
export ROS_DOMAIN_ID=<ID_NUMBER>

Prepare Perception Models#

Important

Review and accept the applicable model licenses before running model setup. The included commands contain license-acceptance flags and can download models. Displaying these instructions does not accept a license or execute installation. Follow your workflow’s model and platform requirements.

FoundationPose estimates the DisplayPort plug and socket poses from RGB-D images, an object mesh, and a segmentation mask. This workflow uses Segment Anything (SAM) to produce masks from the points you select in the camera image. RealSense provides depth directly. Manual-pose insertion does not require these models.

  1. Build isaac_ros_foundationpose, isaac_ros_foundationpose_models_install, and isaac_ros_segment_anything and their dependencies from source in your workspace. Clone isaac_ros_pose_estimation and isaac_ros_image_segmentation if they are not already in ${ISAAC_ROS_WS}/src:

    cd ${ISAAC_ROS_WS}/src
git clone -b release-5.0 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation.git isaac_ros_pose_estimation
git clone -b release-5.0 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation.git isaac_ros_image_segmentation

    Include their Isaac ROS source dependencies in the workspace before resolving system dependencies with rosdep --ignore-src. Run model preparation inside the Isaac ROS environment.

    cd ${ISAAC_ROS_WS}
ISAAC_ROS_SKIP_ASSET_INSTALL=1 MANIPULATOR_INSTALL_ASSETS=0 \
   colcon build --symlink-install --packages-up-to \
   isaac_ros_foundationpose isaac_ros_foundationpose_models_install \
   isaac_ros_segment_anything isaac_ros_manipulation_asset_bringup
source install/setup.bash

  2. Review the FoundationPose model license, then install its refine and score models and generate the TensorRT engines on the deployment machine:

    unset ISAAC_ROS_SKIP_ASSET_INSTALL
ros2 run isaac_ros_foundationpose_models_install install_foundationpose_models.sh --eula

    The workflow expects refine_trt_engine.plan and score_trt_engine.plan under ${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose. Use the plug and socket meshes specified in the DisplayPort tutorial.

  3. Prepare SAM on an x86 machine with the source-built packages above. Review the SAM license before downloading its weights. Install the segment_anything Python package, which the ONNX export requires:

    pip install --no-deps --break-system-packages git+https://github.com/facebookresearch/segment-anything.git@dca509fe793f601edb92606367a655c15ac00fdf

    Then run the targeted setup to download the SAM assets and vit_b weights, export the model to ONNX, and install the Triton configuration:

    MANIPULATOR_INSTALL_ASSETS=1 ros2 run isaac_ros_manipulation_asset_bringup \
   setup_perception_models.py --models segment_anything

    For the individual export steps, refer to Prepare Segment Anything ONNX Model. SAM uses Triton’s ONNX backend. You must run the ONNX conversion on x86.

  4. If you deploy to Jetson, copy the complete ${ISAAC_ROS_WS}/isaac_ros_assets/models/triton/segment_anything directory from the x86 machine to the same workspace-relative location on Jetson. Preserve 1/model.onnx and config.pbtxt.

  5. Configure the DisplayPort workflow to use SEGMENT_ANYTHING for both object_detection_type and segmentation_type, and FOUNDATION_POSE for pose_estimation_type. Set sam_model_repository_paths to ['$ISAAC_ROS_WS/isaac_ros_assets/models/triton']. Wait for Triton to report that segment_anything is loaded before selecting the plug and socket.

Configure Your Robot#

Continue with simulation setup or real-robot setup. If you already built the manipulation packages here, skip that build section in the robot setup guide and continue with its remaining dependencies and configuration.

Choose the model and workflow configuration described in the DisplayPort tutorials.