Setup Hardware and Software for Real Robot with Isaac ROS Manipulation#

Overview#

This tutorial walks through the process of setting up the hardware and software for a real robot with Isaac ROS Manipulation. Isaac ROS Manipulation supports both Universal Robots (UR) and Flexiv Rizon manipulators.

Tutorial#

Set Up Robot, Drivers, and Gripper#

Follow the setup guide for your robot. This includes hardware setup, building the robot driver from source, and configuring the gripper.

Set Up Cameras for Robot#

  1. Refer to the Set Up Cameras for Robot section.

Note

Multiple cameras can help reduce occlusion and noise in the scene and therefore increase the quality and completeness of the 3D reconstruction used for collision avoidance.

While the Pick and Place workflow with multiple cameras runs scene reconstruction for obstacle-aware planning on all cameras, object detection and pose estimation are only enabled on the camera with the lowest index.

Reflective or smooth, featureless surfaces in the environment may increase noise in the depth estimation.

Use of multiple cameras is recommended.

Mixing stereo camera types is untested but may work with modifications to the launch files.

Warning

The obstacle avoidance behavior demonstrated in this tutorial is not a safety function and does not comply with any national or international functional safety standards. When testing obstacle avoidance behavior, do not use human limbs or other living entities.

Configure Docker for Real-Time Scheduling#

If you are using the Isaac ROS Docker container, configure the required Docker arguments for real-time scheduling support. Refer to Configure Docker Arguments for Real-Time Scheduling for instructions.

Set Up Development Environment#

  1. Set up your development environment by following the instructions in getting started.

  2. (Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

    Note

    We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the development environment during installation, which will interrupt the quickstart process.

Build Isaac ROS Manipulation Packages#

  1. Activate the Isaac ROS environment:

    isaac-ros activate

  2. Install and set the environment variable for Cyclone DDS:

    sudo apt-get install -y ros-jazzy-rmw-cyclonedds-cpp
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp

  3. Install system dependencies:

    sudo apt update && sudo apt install -y python3-pip python3-git && pip install --break-system-packages pyvers==0.2.2

  4. Install the prebuilt Debian package:

    sudo apt-get update

    sudo apt-get install -y ros-jazzy-isaac-ros-manipulation-bringup

  1. Clone this repository under ${ISAAC_ROS_WS}/src:

    cd ${ISAAC_ROS_WS}/src && git clone --recursive -b release-4.5 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_manipulation.git isaac_ros_manipulation

  2. Activate the Isaac ROS environment:

    isaac-ros activate

  3. Install and set the environment variable for Cyclone DDS:

    sudo apt-get install -y ros-jazzy-rmw-cyclonedds-cpp
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp

  4. Install system dependencies:

    sudo apt update && sudo apt install -y python3-pip python3-git && pip install --break-system-packages pyvers==0.2.2

  5. Use rosdep to install the package’s dependencies:

    sudo apt-get update

    rosdep update && rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_manipulation/isaac_ros_manipulation_bringup --ignore-src -y

  6. Install the segment_anything package via pip.

    pip install --no-deps --break-system-packages git+https://github.com/facebookresearch/segment-anything.git

  7. Accept NVIDIA model EULAs before building:

    The build process will download perception models (ESS, FoundationStereo, FoundationPose, SyntheticaDETR, Grounding DINO) that require accepting NVIDIA’s End-User License Agreements (EULAs). Set the following environment variable to accept the terms:

    export ISAAC_ROS_ACCEPT_EULA=1

    Note

    By setting this variable, you accept the terms and conditions of the EULAs for the perception models listed above. These models are distributed on the NVIDIA NGC Catalog under NVIDIA’s standard model licenses.

  8. Build the package from source:

    cd ${ISAAC_ROS_WS}
export MANIPULATOR_INSTALL_ASSETS=1
export FOUNDATIONSTEREO_MODEL_RES=low_res
colcon build --symlink-install --packages-up-to isaac_ros_manipulation_bringup

    Note

    Setting FOUNDATIONSTEREO_MODEL_RES=low_res is recommended because the build process installs FoundationStereo models. The default high_res model requires 16 GB of GPU memory during TensorRT conversion, while low_res requires 8 GB.

  9. Source the ROS workspace:

    Note

    Make sure to repeat this step in every terminal created inside the Isaac ROS environment.

    Because this package was built from source, the enclosing workspace must be sourced for ROS to be able to find the package’s contents.

    source install/setup.bash

Set Up Perception Deep Learning Models#

  1. Prepare the ESS model to run depth estimation:

    ros2 run isaac_ros_ess_models_install install_ess_models.sh --eula

  2. The FoundationStereo model is also available for stereo depth estimation. Follow the documentation in FoundationStereo to set up the model in case of any issues.

    export FOUNDATIONSTEREO_MODEL_RES=low_res
ros2 run isaac_ros_foundationstereo_models_install install_foundationstereo_models.sh --eula \
--model_res low_res

  3. Set up the FoundationPose model. Follow the documentation in FoundationPose to set up the model in case of any issues.

    ros2 run isaac_ros_foundationpose_models_install install_foundationpose_models.sh --eula

  4. Set up the SyntheticaDETR model. Follow the documentation in SyntheticaDETR to set up the model in case of any issues.

    ros2 run isaac_ros_rtdetr_models_install install_rtdetr_models.sh --eula

  5. Set up the Grounding DINO model. For troubleshooting, refer to the isaac_ros_grounding_dino package documentation.

    ros2 run isaac_ros_grounding_dino_models_install install_grounding_dino_models.sh --eula

  6. Prepare the Segment Anything (SAM) ONNX model.

    SAM requires converting the PyTorch weights to ONNX format. This conversion can only be performed on x86 machines. If you intend to run on Jetson, you must first perform the conversion on an x86 machine and then copy the generated ONNX files to the Jetson device.

    1. On an x86 machine, install the segment_anything package via pip:

      pip install --no-deps --break-system-packages git+https://github.com/facebookresearch/segment-anything.git

    2. Follow the conversion instructions in the Prepare Segment Anything ONNX Model section of the Segment Anything documentation.

    3. If running on Jetson, copy the generated ONNX model files from the x86 machine to the corresponding location on the Jetson device.

  7. Prepare the Segment Anything 2 (SAM2) ONNX model.

    SAM2 also requires converting the PyTorch weights to ONNX format. This conversion can only be performed on x86 machines. If you intend to run on Jetson, you must first perform the conversion on an x86 machine and then copy the generated ONNX files to the Jetson device.

    1. Follow the conversion instructions in the Prepare Segment Anything2 ONNX Model section of the Segment Anything 2 documentation.

    2. If running on Jetson, copy the generated ONNX model files from the x86 machine to the corresponding location on the Jetson device.

  8. To setup perception models like FoundationPose, SyntheticaDETR and download sample object assets, run the following command. This command will also give users a final verification that all the models are installed correctly:

    export MANIPULATOR_INSTALL_ASSETS=1
export FOUNDATIONSTEREO_MODEL_RES=low_res

ros2 run isaac_ros_manipulation_asset_bringup setup_perception_models.py --models all

    export MANIPULATOR_INSTALL_ASSETS=1
export FOUNDATIONSTEREO_MODEL_RES=low_res

colcon build --packages-up-to isaac_ros_manipulation_asset_bringup

    Note

    Running this command can take up to 15 minutes on Jetson AGX Thor. NVLabs has provided a DOPE pre-trained model using the HOPE dataset. To train your own DOPE model, see here.

    Note

    For details on which models are downloaded and how to set up specific models individually, refer to the isaac ros manipulation asset bringup package documentation.

    Warning

    If you encounter the error ERROR: segment_anything package not found, ensure you have installed the segment_anything package on your x86 machine as described in the SAM model preparation step above.

  9. As a sanity check, run this command to verify that the assets are set up correctly.

    ros2 run isaac_ros_manipulation_asset_bringup setup_perception_models.py --models all

    You should see the following output:

    INFO: === Setting up FoundationPose assets ===
INFO: Mac and Cheese assets already exist at /workspaces/isaac_ros-dev/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1 - Skipping download
INFO: === Setting up DOPE model ===
INFO: DOPE model setup completed successfully
INFO: === Setting up Segment Anything assets ===
INFO: Segment Anything assets already exist at /workspaces/isaac_ros-dev/isaac_ros_assets/isaac_ros_segment_anything - Skipping download
INFO: === Setting up SAM model ===
INFO: SAM model already exists at /workspaces/isaac_ros-dev/isaac_ros_assets/isaac_ros_segment_anything/vit_b.pth - Skipping download
INFO: ONNX model already exists at /workspaces/isaac_ros-dev/isaac_ros_assets/models/triton/segment_anything/1/model.onnx - Skipping conversion
INFO: Config file already exists at /workspaces/isaac_ros-dev/isaac_ros_assets/models/triton/segment_anything/config.pbtxt - Skipping copy
INFO: SAM model setup completed successfully
INFO: === Setting up DNN Policy assets ===
INFO: DNN Policy assets already exist at /workspaces/isaac_ros-dev/isaac_ros_assets/isaac_ros_manipulation_dnn_policy - Skipping download
INFO: Setup results:
INFO: All requested models were set up successfully!