Object Following

Overview

This tutorial demonstrates how to use Isaac for Manipulation to have the robot end effector follow an object at a fixed offset distance while avoiding obstacles detected in the workspace.

For pose estimation, this tutorial provides options to use either FoundationPose or DOPE. This tutorial leverages the cuMotion plugin for MoveIt 2 provided by Isaac ROS cuMotion.

This object-following tutorial supports the same perception models as the one on the real robot.

We use rectified image streams from simulation and do not need to perform lens distortion and rectification that would need to be performed on a real robot.

We use perfect ground truth depth from the simulation which drives up accuracy and allows one to focus on the software stack in a perfect albeit unrealistic environment.

Warning

The obstacle avoidance behavior demonstrated in this tutorial is not a safety function and does not comply with any national or international functional safety standards. When testing obstacle avoidance behavior, do not use human limbs or other living entities.

The example uses RViz for visualization. RViz is the default visualization tool when Isaac for Manipulation is running on the same computer that is displaying the visualization. Foxglove is recommended when visualizing a reconstruction streamed from a remote machine. However, that is not officially supported for the sim workflows.

Prerequisites

• Follow the setup instructions in Setup for Simulated Robot with Isaac for Manipulation in Isaac Sim.

• For improved performance in the scene execution, follow the instructions in Optimization for Physics Scene in Isaac for Manipulation.

Tutorial

Launch Isaac Sim

1. Open Isaac Sim and load the scene:

   https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1/Isaac/Samples/ROS2/Scenario/isaac_manipulator_scene.usd
   

   Tip

   We recommend setting a ROS_DOMAIN_ID via export ROS_DOMAIN_ID=<ID_NUMBER> for every new terminal where you run ROS commands, to avoid interference with other computers in the same network (ROS Guide).

2. Hit Play in Isaac Sim to start the simulation.

Run the Object Following Example

1. Open a new terminal and activate the Isaac ROS environment:

   isaac-ros activate
   

2. Configure the workflow based on your chosen perception model:

   Object Following using FoundationPoseObject Following using DOPE

   1. Change workflow_type in the configuration file to OBJECT_FOLLOWING. The configuration file path is:

      ${ISAAC_ROS_WS}/src/isaac_manipulator/isaac_manipulator_bringup/params/sim_launch_params.yaml
      

      The manipulator_workflow_config parameter points to the manipulator configuration file. Please refer to Create Manipulator Configuration File section.

   2. Change pose_estimation_type in the configuration file to FOUNDATION_POSE

3. Launch the object following example:

   ros2 launch isaac_manipulator_bringup workflows.launch.py \
      manipulator_workflow_config:=$(ros2 pkg prefix --share isaac_manipulator_bringup)/params/sim_launch_params.yaml
   

   The manipulator_workflow_config parameter points to the manipulator configuration file. Please refer to Create Manipulator Configuration File section.

Next Steps

• Try the Pick and Place tutorial for complete manipulation workflows

• Try the Pose to Pose tutorial for simpler motion planning

• Learn more about Isaac for Manipulation Reference Architecture