Set Up Hardware and Software for a Real Robot with Isaac ROS Manipulation#

Overview#

This tutorial sets up the hardware and software for a real robot with Isaac ROS Manipulation. This guide covers the Flexiv Rizon robot and Grav gripper used for DisplayPort insertion.

Tutorial#

Set Up Development Environment#

Complete Workspace and Developer Environment Setup on the robot deployment device and any model preparation host before building the robot packages. The simulation tutorial shares this prerequisite.
Install dependencies for any sensors you want to use by following the sensor-specific guides.

Note

If you are using the Isaac ROS Docker container, configure the required Docker arguments for real-time scheduling support. Refer to Configure Docker Arguments for Real-Time Scheduling for instructions.

Set Up Robot, Drivers, and Gripper#

Follow the setup guide for your robot. This includes hardware setup, building the robot driver from source, and configuring the gripper.

Refer to the Flexiv Robots setup guide, which covers:

Setting up the Flexiv Rizon robot for remote control
Building Flexiv drivers from source
Configuring the Grav gripper
Syncing robot-specific kinematics

Set Up Cameras for the Robot#

Refer to the camera configuration guide.

#. For perception-derived poses, use a calibrated RealSense camera with a clear view of the DisplayPort plug and socket. Check the RGB and depth images before running #. For perception-derived poses, use a calibrated RealSense camera with a clear view of the DisplayPort plug and socket. Check the RGB and depth images before running FoundationPose and Segment Anything (SAM). The manual-pose deployment path uses measured poses.

Build Isaac ROS Manipulation Packages#

For the physical-robot workflow, configure Zenoh before building:

Install Zenoh and set its environment variable:

sudo apt-get install -y ros-lyrical-rmw-zenoh-cpp
source /opt/ros/lyrical/setup.bash
export RMW_IMPLEMENTATION=rmw_zenoh_cpp

Note

Zenoh requires a router daemon. Start it once, in its own terminal, before launching anything else, and leave it running:

export RMW_IMPLEMENTATION=rmw_zenoh_cpp
ros2 run rmw_zenoh_cpp rmw_zenohd

Build the manipulation packages and their Isaac ROS dependencies from source. Place the dependency repositories in ${ISAAC_ROS_WS}/src before running rosdep so --ignore-src resolves those packages from the workspace. The dependency installation commands below provide system and ROS prerequisites.

Clone the isaac_ros_manipulation repository under ${ISAAC_ROS_WS}/src:

cd ${ISAAC_ROS_WS}/src && git clone --recursive -b release-r1 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_manipulation.git isaac_ros_manipulation

Activate the Isaac ROS environment:
```
isaac-ros activate
```

Install system dependencies:

sudo apt update && sudo apt install -y python3-pip python3-git && pip install --break-system-packages pyvers==0.2.2

Use rosdep to install the package’s dependencies:

sudo apt-get update
rosdep update && rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_manipulation/isaac_ros_manipulation_bringup --ignore-src -y

Build the package from source, with automatic asset installation disabled. Prepare any perception models required by your workflow separately.

cd ${ISAAC_ROS_WS}
ISAAC_ROS_SKIP_ASSET_INSTALL=1 MANIPULATOR_INSTALL_ASSETS=0 colcon build --symlink-install --packages-up-to isaac_ros_manipulation_bringup

Source the ROS workspace and set your ROS_DOMAIN_ID:

Note

Repeat this step in every terminal that you open inside the Isaac ROS environment.

Because you built this package from source, you must source the enclosing workspace so that ROS can find the package’s contents.
```
source install/setup.bash
export ROS_DOMAIN_ID=<ID_NUMBER>
```

Set Up Perception Deep Learning Models#

FoundationPose estimates the DisplayPort plug and socket poses from RGB-D images, an object mesh, and a segmentation mask. This workflow uses Segment Anything (SAM) to produce masks from the points you select in the camera image. RealSense provides depth directly. Manual-pose insertion does not require these models.

Build isaac_ros_foundationpose, isaac_ros_foundationpose_models_install, and isaac_ros_segment_anything and their dependencies from source in your workspace. Clone isaac_ros_pose_estimation and isaac_ros_image_segmentation if they are not already in ${ISAAC_ROS_WS}/src:

cd ${ISAAC_ROS_WS}/src
git clone -b release-5.0 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation.git isaac_ros_pose_estimation
git clone -b release-5.0 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation.git isaac_ros_image_segmentation

Include their Isaac ROS source dependencies in the workspace before resolving system dependencies with rosdep --ignore-src. Run model preparation inside the Isaac ROS environment.

cd ${ISAAC_ROS_WS}
ISAAC_ROS_SKIP_ASSET_INSTALL=1 MANIPULATOR_INSTALL_ASSETS=0 \
   colcon build --symlink-install --packages-up-to \
   isaac_ros_foundationpose isaac_ros_foundationpose_models_install \
   isaac_ros_segment_anything isaac_ros_manipulation_asset_bringup
source install/setup.bash

Review the FoundationPose model license, then install its refine and score models and generate the TensorRT engines on the deployment machine:
```
unset ISAAC_ROS_SKIP_ASSET_INSTALL
ros2 run isaac_ros_foundationpose_models_install install_foundationpose_models.sh --eula
```
The workflow expects refine_trt_engine.plan and score_trt_engine.plan under ${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose. Use the plug and socket meshes specified in the DisplayPort tutorial.
Prepare SAM on an x86 machine with the source-built packages above. Review the SAM license before downloading its weights. Install the segment_anything Python package, which the ONNX export requires:
```
pip install --no-deps --break-system-packages git+https://github.com/facebookresearch/segment-anything.git@dca509fe793f601edb92606367a655c15ac00fdf
```
Then run the targeted setup to download the SAM assets and vit_b weights, export the model to ONNX, and install the Triton configuration:
```
MANIPULATOR_INSTALL_ASSETS=1 ros2 run isaac_ros_manipulation_asset_bringup \
   setup_perception_models.py --models segment_anything
```
For the individual export steps, refer to Prepare Segment Anything ONNX Model. SAM uses Triton’s ONNX backend. You must run the ONNX conversion on x86.
If you deploy to Jetson, copy the complete ${ISAAC_ROS_WS}/isaac_ros_assets/models/triton/segment_anything directory from the x86 machine to the same workspace-relative location on Jetson. Preserve 1/model.onnx and config.pbtxt.
Configure the DisplayPort workflow to use SEGMENT_ANYTHING for both object_detection_type and segmentation_type, and FOUNDATION_POSE for pose_estimation_type. Set sam_model_repository_paths to ['$ISAAC_ROS_WS/isaac_ros_assets/models/triton']. Wait for Triton to report that segment_anything is loaded before selecting the plug and socket.

Run Preflight Tests on the Real Robot#

These tests are optional but highly recommended. Follow the Flexiv driver and DisplayPort checks. Use your edited DisplayPort configuration and run only the checks for your setup.