Setup for Simulated Robot in Isaac Sim#

Overview#

This guide walks through the process of setting up the software for a simulated robot with Isaac for Manipulation in Isaac Sim.

Requirements#

  • Install Isaac Sim following the instructions in Isaac ROS Isaac Sim Setup Guide.

  • Isaac Sim running on a workstation that has at least 25 GB GPU VRAM and 30 GB CPU RAM. In case of questions, refer to the official Isaac Sim requirement documentation.

Set Up Development Environment#

  1. Set up your development environment by following the instructions in getting started.

Build Isaac for Manipulation Packages#

  1. Activate the Isaac ROS environment:

    isaac-ros activate

  2. Install the prebuilt Debian package:

    sudo apt-get update

    sudo apt-get install -y ros-jazzy-isaac-manipulator-bringup

  1. Clone this repository under ${ISAAC_ROS_WS}/src:

    cd ${ISAAC_ROS_WS}/src && git clone --recursive -b release-4.0 https://github.com/NVIDIA-ISAAC-ROS/isaac_manipulator.git isaac_manipulator

  2. Activate the Isaac ROS environment:

    isaac-ros activate

  3. Use rosdep to install the package’s dependencies:

    sudo apt-get update

    rosdep update && rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_manipulator/isaac_manipulator_bringup --ignore-src -y

  4. Install the segment_anything package via pip.

    pip install --no-deps --break-system-packages git+https://github.com/facebookresearch/segment-anything.git

  5. Build the package from source:

    cd ${ISAAC_ROS_WS}
export MANIPULATOR_INSTALL_ASSETS=1
colcon build --symlink-install --packages-up-to isaac_manipulator_bringup

  6. Source the ROS workspace:

    Note

    Make sure to repeat this step in every terminal created inside the Isaac ROS environment.

    Because this package was built from source, the enclosing workspace must be sourced for ROS to be able to find the package’s contents.

    source install/setup.bash

Note

  • In all terminals, set the following flag for proper communication.

export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp

  • If you are running any other workflow apart from GEAR_ASSEMBLY, then change the PhysicsScene prim by going to its properties and changing the Time Step Per Second to 30 or 60. This will make the scene run significantly faster. Gear Assembly needs a time step of 120 because of the higher quality of physics simulation required.

Setup Perception Models#

  1. Set up the perception models by following the instructions in Set Up Perception Deep Learning Models section.

  1. Clone the Isaac ROS fork of ros2_robotiq_gripper and tylerjw/serial under ${ISAAC_ROS_WS}/src:

    cd ${ISAAC_ROS_WS}/src && \
  git clone --recursive https://github.com/NVIDIA-ISAAC-ROS/ros2_robotiq_gripper && \
  git clone -b ros2 https://github.com/tylerjw/serial

    Note

    • The fork is used to fix this bug in the original repository.

    • The custom serial package build is required because of Issue 21

  2. Build the gripper dependencies:

    cd ${ISAAC_ROS_WS}
colcon build --symlink-install --packages-select-regex robotiq* serial --cmake-args "-DBUILD_TESTING=OFF" && \
source install/setup.bash  # Source the workspace after building gripper dependencies

Install Python Dependencies (only required for workflows that run RL policies)#

Install RSL-RL:

sudo apt-get install -y python3-git \
   && pip install --break-system-packages tensordict \
   && pip install --break-system-packages --no-deps rsl-rl-lib==3.1.1

Prepare Configuration Files and Environment Variables#

Before editing any configuration files, copy them to the appropriate location and set up environment variables based on your installation method.

  1. Binary installations have read-only configuration files in system directories. Copy all necessary files to a writable location:

    # Create a directory for your custom configuration
mkdir -p ${ISAAC_ROS_WS}/isaac_manipulator_config

  2. Copy the workflow configuration file for your robot:

    cp $(ros2 pkg prefix --share isaac_manipulator_bringup)/params/sim_launch_params.yaml \
      ${ISAAC_ROS_WS}/isaac_manipulator_config/my_robot_config.yaml

  3. Set up environment variables:

    # Point to the directory containing your configuration files
export ISAAC_MANIPULATOR_WORKFLOW_CONFIG_DIR="${ISAAC_ROS_WS}/isaac_manipulator_config"

  1. When building from source with --symlink-install, you can edit configuration files directly in the source directories.

  2. Set up environment variables:

    # Point to the source directories containing the configuration files
export ISAAC_MANIPULATOR_WORKFLOW_CONFIG_DIR="${ISAAC_ROS_WS}/src/isaac_manipulator/isaac_manipulator_bringup/params"

Launch Isaac Sim#

  1. Open Isaac Sim and load the scene:

    https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1/Isaac/Samples/ROS2/Scenario/isaac_manipulator_scene.usd

    Tip

    We recommend setting a ROS_DOMAIN_ID via export ROS_DOMAIN_ID=<ID_NUMBER> for every new terminal where you run ROS commands, to avoid interference with other computers in the same network (ROS Guide).

  2. Hit Play in Isaac Sim to start the simulation.

Optimization for Physics Scene (Optional)#

  1. If you are running the gear assembly workflow, please skip this section.

  2. Gear Assembly needs a time step of 120 and GPU physics because of the higher quality of physics simulation required.

  1. Stop the scene.

  2. Disable the Enable GPU dynamics option in the PhysicsScene prim. This will make the scene run significantly faster. When we have a scene like the one in this tutorial which has a single robot, CPU physics generally runs much faster.

  3. Change the PhysicsScene prim (/World/PhysicsScene) by going to its properties and changing the Time Step Per Second to 30 or 60. This will make the scene run significantly faster. Gear Assembly needs a time step of 120 because of the higher quality of physics simulation required.

    PhysicsScene

  4. Start the scene.