isaac_ros_moveit_benchmark

Overview

Isaac ROS MoveIt Benchmark consists of utility packages to measure the performance of various MoveIt 2 plugins, including Isaac ROS cuMotion, on various motion planning problems. Metrics related to planner performance, such as motion time and planning time, as well as metrics related to system usage, such as CPU usage are computed to provide a comprehensive view of planner performance.

The problems are sourced from the MotionBenchmarkMaker (MBM) dataset. Currently, the benchmark supports the UR5 CB3 robot and the Franka robot.

The isaac_ros_moveit_benchmark provides a standardized, convenient method to benchmark various planners that are available as MoveIt 2 plugins.

To see how isaac_ros_moveit_benchmark is used to benchmark some specific planners, please consult the following packages:

• UR5:

  • UR5 cuMotion Benchmark

  • UR5 OMPL Benchmark

• Franka:

  • Franka cuMotion Benchmark

  • Franka OMPL Benchmark

Setup

Set Up Development Environment

1. Set up your development environment by following the instructions in Getting Started.

2. Clone isaac_ros_common under ${ISAAC_ROS_WS}/src.

   cd ${ISAAC_ROS_WS}/src && \
      git clone -b release-3.2 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common.git isaac_ros_common
   

Build the Benchmarks

Binary PackageBuild from Source

1. Launch the Docker container using the run_dev.sh script:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh
   

2. Install the prebuilt Debian package:

   sudo apt-get update
   

   sudo apt-get install -y ros-humble-isaac-ros-ur5-cumotion-benchmark ros-humble-isaac-ros-ur5-ompl-benchmark ros-humble-isaac-ros-franka-cumotion-benchmark ros-humble-isaac-ros-franka-ompl-benchmark
   

Running the Benchmark

UR5 cuMotionUR5 OMPLFranka cuMotionFranka OMPL

launch_test $(ros2 pkg prefix isaac_ros_ur5_cumotion_benchmark)/share/isaac_ros_ur5_cumotion_benchmark/ur5_cumotion_benchmark_scripts/ur5_cumotion_mbm.py

Available Metrics

Metric Name	Explanation	Interpretation
Success Rate	The percentage of successful plans over all problems	A higher success rate is better, as it implies the planner can work across a variety of scenarios.
Planning Time	The time taken by the planner to compute a valid plan.	A lower planning time is better, as it implies the planner can generate plans faster.
Motion Time	The time it would take the robot to execute the plan generated by the planner.	A lower motion time is better, as a robot can execute the plan faster.
Path Length	The total path length of the plan in radians (as measured in joint space, using an unweighted L2 norm)	A lower path length is better, as it means the robot needs to move less to execute the plan.
Jerk	The maximum jerk along the entire planned path	A lower jerk is better, as it means the path is more smooth, enabling a robot to execute it more easily.