isaac_ros_moveit_benchmark

Overview

Isaac ROS MoveIt Benchmark consists of utility packages to measure the performance of various MoveIt 2 plugins, including Isaac ROS cuMotion, on various motion planning problems. Metrics related to planner performance, such as motion time and planning time, as well as metrics related to system usage, such as CPU usage are computed to provide a comprehensive view of planner performance.

The problems are sourced from the MotionBenchmarkMaker (MBM) dataset. The benchmark supports the UR5 CB3 robot and the Franka robot.

The isaac_ros_moveit_benchmark provides a standardized, convenient method to benchmark various planners that are available as MoveIt 2 plugins.

To see how isaac_ros_moveit_benchmark is used to benchmark some specific planners, consult the following packages:

• UR5:

  • UR5 cuMotion Benchmark

  • UR5 OMPL Benchmark

• Franka:

  • Franka cuMotion Benchmark

  • Franka OMPL Benchmark

Setup

Set Up Development Environment

1. Set up your development environment by following the instructions in getting started.

2. (Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

   Note

   We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the development environment during installation, which will interrupt the quickstart process.

Build the Benchmarks

Binary PackageBuild from Source

1. Activate the Isaac ROS environment:

   isaac-ros activate
   

2. Install the prebuilt Debian package:

   sudo apt-get update
   

   sudo apt-get install -y ros-jazzy-isaac-ros-ur5-cumotion-benchmark ros-jazzy-isaac-ros-ur5-ompl-benchmark ros-jazzy-isaac-ros-franka-cumotion-benchmark ros-jazzy-isaac-ros-franka-ompl-benchmark
   

Running the Benchmark

UR5 cuMotionUR5 OMPLFranka cuMotionFranka OMPL

launch_test $(ros2 pkg prefix isaac_ros_ur5_cumotion_benchmark)/share/isaac_ros_ur5_cumotion_benchmark/ur5_cumotion_benchmark_scripts/ur5_cumotion_mbm.py

Interpreting Benchmark Results

MoveIt planner benchmarks report planner metrics rather than frame-rate metrics. The primary validation value is the planner’s success rate. In the console report this is shown as Planning Success Rate (%). In the exported JSON report the same metric appears under the planner name as PlannerPerformanceMetrics.PLANNING_SUCCESS_RATE.

Use Mean Planning Time (ms) / PlannerPerformanceMetrics.MEAN_PLANNING_TIME as the primary timing metric when comparing planner performance. Motion time, path length, jerk, and CPU/GPU resource metrics are supplemental metrics for comparing trajectory quality and system load. The MBM benchmark scripts also include per_problem_metrics in the JSON report so individual results can be inspected in dataset order during validation.

For example, a cuMotion MBM report is structured like this:

{
  "cuMotion": {
    "PlannerPerformanceMetrics.PLANNING_SUCCESS_RATE": 100.0,
    "PlannerPerformanceMetrics.MEAN_PLANNING_TIME": 12.3,
    "PlannerPerformanceMetrics.MEAN_MOTION_TIME": 1.7,
    "PlannerPerformanceMetrics.MEAN_PATH_LENGTH": 2.4,
    "PlannerPerformanceMetrics.MEAN_JERK": 42.0,
    "per_problem_metrics": [
      {
        "planning_time": 11.8,
        "motion_time": 1.6,
        "path_length": 2.3,
        "max_jerk": 40.0
      }
    ]
  },
  "metadata": {
    "BenchmarkMetadata.NAME": "Isaac ROS cuMotion Node MBM Benchmark"
  }
}

Do not expect BasicPerformanceMetrics.MEAN_FRAME_RATE_(FPS) or playback FPS fields from these scripts. Those fields are produced by graph throughput benchmarks that replay message streams; the UR5 and Franka MoveIt benchmarks send motion-planning requests and validate the returned plans.

Available Metrics

Metric Name	Explanation	Interpretation
Success Rate	The percentage of successful plans over all problems.	A higher success rate is better, because it indicates that the planner can work across a variety of scenarios.
Planning Time	The time taken by the planner to compute a valid plan.	A lower planning time is better, because it indicates that the planner can generate plans faster.
Motion Time	The time it would take the robot to execute the plan generated by the planner.	A lower motion time is better, because a robot can execute the plan faster.
Path Length	The total path length of the plan in radians (as measured in joint space, using an unweighted L2 norm).	A lower path length is better, because it means that the robot needs to move less to execute the plan.
Jerk	The maximum jerk along the planned trajectory.	A lower jerk is better, because it means that the trajectory is smoother and can be tracked more easily by the robot controller.