Tutorial for Training and Deploying an RL Policy from Isaac Lab to a Real Robot

Overview

This tutorial guides you through the process of deploying a basic RL reach policy trained in Isaac Lab on a real manipulator using Isaac ROS. Both UR10e and Flexiv Rizon 4s robots are supported.

Prerequisites

• Follow the setup instructions in Setup Hardware and Software for Real Robot. Make sure to complete the tab corresponding to your robot (UR or Flexiv) in each step.

Tutorial

Train and Validate Policy using Isaac Lab

1. Set up Isaac Lab using the local installation, but clone the following fork and branch instead of the default Isaac Lab repository:

   git clone -b isaac_ros_training https://github.com/shauryadNv/IsaacLab.git
   

   Then continue with the rest of the installation steps from within the cloned IsaacLab directory.

   Note

   The training environment for this task uses the PhysX backend and requires a full Isaac Sim installation. The kit-less (standalone) setup of Isaac Lab is not sufficient. Make sure to follow the installation method that includes Isaac Sim.

2. Train a policy with RSL-RL:

   UR RobotFlexiv Robot

   ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py \
      --task Isaac-Deploy-Reach-UR10e-ROS-Inference-v0 --headless
   

3. Validate the policy in simulation:

   UR RobotFlexiv Robot

   ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/play.py \
      --task Isaac-Deploy-Reach-UR10e-ROS-Inference-v0 --num_envs 1 --checkpoint <CHECKPOINT> \
      --visualizer kit
   

   Replace <CHECKPOINT> with the path to the .pth policy checkpoint file.

Deploy the Policy using Isaac ROS

1. Install Cyclone DDS (one-time setup):

   sudo apt-get install -y ros-jazzy-rmw-cyclonedds-cpp
   

2. Set the Cyclone DDS environment variable (required in every terminal):

   export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
   

3. Set up your development environment using the instructions in getting started.

4. Install isaac_ros_manipulation_dnn_policy

   Binary PackageBuild from Source

   1. Activate the Isaac ROS environment:

      isaac-ros activate
      

   2. Install the prebuilt Debian package:

      sudo apt-get update
      

      sudo apt-get install -y ros-jazzy-isaac-ros-manipulation-dnn-policy
      

5. Open a new terminal, activate the Isaac ROS environment, and set the Cyclone DDS environment variable:

   isaac-ros activate
   export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
   

6. Start the robot driver:

   UR RobotFlexiv Robot

   ros2 launch ur_robot_driver ur_control.launch.py ur_type:=<UR_TYPE> robot_ip:=<ROBOT_IP> initial_joint_controller:=impedance_controller launch_rviz:=False kinematics_params_file:=<calibration_file_path>
   

   Replace <UR_TYPE> with the type of your UR robot (for example, ur10e) and <ROBOT_IP> with the IP address of your robot.

   Note

   Replace <calibration_file_path> with the path to the calibration file for your robot. To calibrate your UR robot, refer to the ur_calibration usage.

7. In a separate terminal, activate the Isaac ROS environment, set the Cyclone DDS environment variable, and run the inference pipeline:

   isaac-ros activate
   export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
   

   UR RobotFlexiv Robot

   ros2 launch isaac_ros_manipulation_dnn_policy inference.launch.py checkpoint:=<CHECKPOINT>
   

   Replace <CHECKPOINT> with the path to the .pth policy checkpoint file.

8. In a separate terminal, activate the Isaac ROS environment and set the Cyclone DDS environment variable:

   isaac-ros activate
   export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
   

9. Set a target end-effector pose:

   POSITION="{x: <X>, y: <Y>, z: <Z>}"; ORIENTATION="{w: <QW>, x: <QX>, y: <QY>, z: <QZ>}"
   

   Replace <X>, <Y>, and <Z> with the target end-effector position and <QW>, <QX>, <QY>, and <QZ> with the target end-effector orientation as quaternion values.

10. Publish the target end-effector pose.

    Note

    The frame_id differs between robots: UR uses base while Flexiv uses world as the reference frame for the goal pose.

    UR RobotFlexiv Robot

    ros2 topic pub -r 60 /goal_pose geometry_msgs/msg/PoseStamped \
       "{ header: { stamp: now, frame_id: base }, pose: { position: $POSITION, orientation: $ORIENTATION } }"
    

11. Enable the robot controller:

    UR RobotFlexiv Robot

    On the UR teach pendant, press play to enable the robot.

Troubleshooting

Out of distribution start state of the robot before reach

The policy for reach is trained with only a specific number of Inverse Kinematics (IK) starting solutions.

The other thing to make sure is that the pose we are sending to the reach policy is also in distribution. We make sure of this by having a check at the policy level where before any inference, we check if the pose is in distribution. If you see a log message like the one shown below, then the pose is not in distribution and the reach policy is unlikely to work.

[WARNING] [observation_encoder_node]: target position out of distribution

It will also mention the bounds of the pose that are acceptable, if you want to change this distribution please check the target_pos_centre, target_pos_range, target_rot_centre, target_rot_range flags in the Isaac Lab documentation during policy training.

Conclusion

So in essence, the user must make sure to do the following things:

• Ensure the robot is not in an OOD start state

• Ensure the pose we are sending to the reach policy is in distribution

If you have verified all of the above, then the reach policy is likely to work.