`isaac_ros_data_replayer`

Source code on GitHub.

Overview

The isaac_ros_data_replayer package enables replaying sensor data recorded using the isaac_ros_data_recorder package. This package also enables reliable extraction of stereo cameras in different formats, including depth using AI-based perception. Supported formats include MP4, PNG, and PFM. Stereo camera frames are extracted in synchronized pairs while depth frames are aligned to the left stereo imager.

Quickstart

Set Up Development Environment

Set up your development environment by following the instructions in getting started.

Clone isaac_ros_common under ${ISAAC_ROS_WS}/src.

cd ${ISAAC_ROS_WS}/src && \
   git clone -b release-3.1 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common.git isaac_ros_common

(Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

Warning

We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the Isaac ROS Dev container during installation, which will interrupt the quickstart process.

Download Quickstart Assets

Download quickstart data from NGC:

Make sure required libraries are installed.

sudo apt-get install -y curl jq tar

Then, run these commands to download the asset from NGC:

NGC_ORG="nvidia"
NGC_TEAM="isaac"
PACKAGE_NAME="isaac_ros_data_replayer"
NGC_RESOURCE="isaac_ros_data_replayer_assets"
NGC_FILENAME="quickstart.tar.gz"
MAJOR_VERSION=3
MINOR_VERSION=1
VERSION_REQ_URL="https://catalog.ngc.nvidia.com/api/resources/versions?orgName=$NGC_ORG&teamName=$NGC_TEAM&name=$NGC_RESOURCE&isPublic=true&pageNumber=0&pageSize=100&sortOrder=CREATED_DATE_DESC"
AVAILABLE_VERSIONS=$(curl -s \
    -H "Accept: application/json" "$VERSION_REQ_URL")
LATEST_VERSION_ID=$(echo $AVAILABLE_VERSIONS | jq -r "
    .recipeVersions[]
    | .versionId as \$v
    | \$v | select(test(\"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+$\"))
    | split(\".\") | {major: .[0]|tonumber, minor: .[1]|tonumber, patch: .[2]|tonumber}
    | select(.major == $MAJOR_VERSION and .minor <= $MINOR_VERSION)
    | \$v
    " | sort -V | tail -n 1
)
if [ -z "$LATEST_VERSION_ID" ]; then
    echo "No corresponding version found for Isaac ROS $MAJOR_VERSION.$MINOR_VERSION"
    echo "Found versions:"
    echo $AVAILABLE_VERSIONS | jq -r '.recipeVersions[].versionId'
else
    mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets && \
    FILE_REQ_URL="https://api.ngc.nvidia.com/v2/resources/$NGC_ORG/$NGC_TEAM/$NGC_RESOURCE/\
versions/$LATEST_VERSION_ID/files/$NGC_FILENAME" && \
    curl -LO --request GET "${FILE_REQ_URL}" && \
    tar -xf ${NGC_FILENAME} -C ${ISAAC_ROS_WS}/isaac_ros_assets && \
    rm ${NGC_FILENAME}
fi

Build `isaac_ros_data_replayer`

Launch the Docker container using the run_dev.sh script:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
  ./scripts/run_dev.sh

Install the prebuilt Debian package:

sudo apt-get install -y ros-humble-isaac-ros-data-replayer
source /opt/ros/humble/setup.bash

Install the required assets:

sudo apt-get install -y ros-humble-isaac-ros-ess-models-install
source /opt/ros/humble/setup.bash

ros2 run isaac_ros_ess_models_install install_ess_models.sh --eula

Clone this repository under ${ISAAC_ROS_WS}/src:

cd ${ISAAC_ROS_WS}/src && \
   git clone -b release-3.1 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nova.git isaac_ros_nova

Launch the Docker container using the run_dev.sh script:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh

Use rosdep to install the package’s dependencies:

rosdep install -i -r --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_nova/isaac_ros_data_replayer/ --rosdistro humble -y

Install the required assets:

sudo apt-get install -y ros-humble-isaac-ros-ess-models-install
source /opt/ros/humble/setup.bash

ros2 run isaac_ros_ess_models_install install_ess_models.sh --eula

Build the ROS package in the Docker container:

colcon build --symlink-install --packages-up-to isaac_ros_data_replayer
source install/setup.bash

Run Launch File

ros2 launch isaac_ros_data_replayer data_replayer.launch.py \
   rosbag:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_data_replayer/quickstart

Replay a single camera

ros2 launch isaac_ros_data_replayer data_replayer.launch.py \
   rosbag:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_data_replayer/quickstart \
   enabled_stereo_cameras:=front_stereo_camera enabled_fisheye_cameras:=None enable_3d_lidar:=False

Visualize

Open a new terminal inside the Docker container:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh

Run RViz to visualize data:
```
rviz2
```

Data Extraction

Run data extraction:

ros2 run isaac_ros_data_replayer data_extraction.py \
   --rosbag ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_data_replayer/quickstart \
   --camera front_stereo_camera --output ${ISAAC_ROS_WS} \
   --engine_file_path ${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.0.0/ess.engine

ros2 run isaac_ros_data_replayer data_extraction.py \
   --rosbag ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_data_replayer/quickstart \
   --camera front_stereo_camera --output ${ISAAC_ROS_WS} \
   --engine_file_path ${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.0.0/light_ess.engine

Convert rosbag to MP4:

ros2 run isaac_ros_data_replayer camera_converter.py \
   -i  ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_data_replayer/quickstart \
   -t /front_stereo_camera/left/image_compressed \
   -o ${ISAAC_ROS_WS}/front_stereo_camera_left.mp4

Convert MP4 to PNG:

mkdir ${ISAAC_ROS_WS}/front_stereo_camera_left && \
   ffmpeg -i ${ISAAC_ROS_WS}/front_stereo_camera_left.mp4 -vf fps=30 ${ISAAC_ROS_WS}/front_stereo_camera_left/%05d.png

Troubleshooting

Isaac ROS Troubleshooting

For solutions to problems with Isaac ROS, see troubleshooting.

Not all camera frames extracted from recording

Data extraction extracts synchronized camera frames from a recording. If the number of frames from the left and right camera are not equal, then some frames will be ignored to keep the left and right cameras in sync.

API

Usage

ros2 launch isaac_ros_data_replayer data_replayer.launch.py rosbag:=<rosbag>

ROS Launch Arguments

ROS Launch Argument	Default Value	Description
`rosbag`	None	Path to recording.
`enabled_stereo_cameras`	`front_stereo_camera` `back_stereo_camera` `left_stereo_camera` `right_stereo_camera`	Stereo cameras enabled for replay.
`enabled_fisheye_cameras`	`front_fisheye_camera` `back_fisheye_camera` `left_fisheye_camera` `right_fisheye_camera`	Fisheye cameras enabled for replay.
`enable_3d_lidar`	`True`	Enable 3D lidar for replay.
`replay_loop`	`False`	Loop replay.
`replay_rate`	`1.0`	Extraction rate; 1.0 for real-time, <1.0 for sub-real-time, >1.0 for super-real-time.
`replay_delay`	None	Delay to start replay.
`replay_additional_args`	`--disable-keyboard-controls`	Additional arguments for replay.

ROS Topics Published

ROS Topic	Interface	Description
`/rosout`	rcl_interfaces/msg/Log	Console logs.
`/tf`	tf2_msgs/msg/TFMessage	Movable transforms on the robot.
`/tf_static`	tf2_msgs/msg/TFMessage	Fixed transforms on the robot.
`/front_stereo_camera/left/image_raw`	sensor_msgs/msg/Image	Front stereo camera left camera stream.
`/front_stereo_camera/left/camera_info`	sensor_msgs/msg/CameraInfo	Front stereo camera left camera intrinsics.
`/front_stereo_camera/right/image_raw`	sensor_msgs/msg/Image	Front stereo camera right camera stream.
`/front_stereo_camera/right/camera_info`	sensor_msgs/msg/CameraInfo	Front stereo camera right camera intrinsics.
`/back_stereo_camera/left/image_raw`	sensor_msgs/msg/Image	Back stereo camera left camera stream.
`/back_stereo_camera/left/camera_info`	sensor_msgs/msg/CameraInfo	Back stereo camera left camera intrinsics.
`/back_stereo_camera/right/image_raw`	sensor_msgs/msg/Image	Back stereo camera right camera stream.
`/back_stereo_camera/right/camera_info`	sensor_msgs/msg/CameraInfo	Back stereo camera right camera intrinsics.
`/left_stereo_camera/left/image_raw`	sensor_msgs/msg/Image	Left stereo camera left camera stream.
`/left_stereo_camera/left/camera_info`	sensor_msgs/msg/CameraInfo	Left stereo camera left camera intrinsics.
`/left_stereo_camera/right/image_raw`	sensor_msgs/msg/Image	Left stereo camera right camera stream.
`/left_stereo_camera/right/camera_info`	sensor_msgs/msg/CameraInfo	Left stereo camera right camera intrinsics.
`/right_stereo_camera/left/image_raw`	sensor_msgs/msg/Image	Right stereo camera left camera stream.
`/right_stereo_camera/left/camera_info`	sensor_msgs/msg/CameraInfo	Right stereo camera left camera intrinsics.
`/right_stereo_camera/right/image_raw`	sensor_msgs/msg/Image	Right stereo camera right camera stream.
`/right_stereo_camera/right/camera_info`	sensor_msgs/msg/CameraInfo	Right stereo camera right camera intrinsics.
`/front_fisheye_camera/left/image_raw`	sensor_msgs/msg/Image	Front fisheye camera stream.
`/front_fisheye_camera/left/camera_info`	sensor_msgs/msg/CameraInfo	Front fisheye camera intrinsics.
`/back_fisheye_camera/left/image_raw`	sensor_msgs/msg/Image	Back fisheye camera stream.
`/back_fisheye_camera/left/camera_info`	sensor_msgs/msg/CameraInfo	Back fisheye camera intrinsics.
`/left_fisheye_camera/left/image_raw`	sensor_msgs/msg/Image	Left fisheye camera stream.
`/left_fisheye_camera/left/camera_info`	sensor_msgs/msg/CameraInfo	Left fisheye camera intrinsics.
`/right_fisheye_camera/left/image_raw`	sensor_msgs/msg/Image	Right fisheye camera stream.
`/right_fisheye_camera/left/camera_info`	sensor_msgs/msg/CameraInfo	Right fisheye camera intrinsics.
`/front_2d_lidar/scan`	sensor_msgs/msg/LaserScan	Front 2D lidar scan.
`/back_2d_lidar/scan`	sensor_msgs/msg/LaserScan	Back 2D lidar scan.
`/front_3d_lidar/lidar_points`	sensor_msgs/msg/PointCloud2	Front 3D lidar point cloud.
`/front_stereo_imu/imu`	sensor_msgs/msg/Imu	Front stereo camera inertial measurement unit.
`/chassis/imu`	sensor_msgs/msg/Imu	Chassis inertial measurement unit.
`/chassis/ticks`	isaac_ros_nova_interfaces/msg/EncoderTicks	Chassis encoder count.
`/chassis/odom`	nav_msgs/msg/Odometry	Chassis odometry.
`/chassis/battery_state`	sensor_msgs/msg/BatteryState	Chassis battery state.

Usage

ros2 run isaac_ros_data_replayer data_extraction.py --rosbag <rosbag> --camera <camera> --output <output>

ROS Run Arguments

ROS Run Argument	Default Value	Description
`--rosbag`	None	Path to recording.
`--camera`	None	Camera to extract i.e. `front_stereo_camera`.
`--output`	None	Directory to store extracted data.
`--output_width`	`960`	Rectified image width.
`--output_height`	`576`	Rectified image height.
`--min_disparity`	`3.0`	Minimum disparity value for normalization.
`--max_disparity`	`200.0`	Maximum disparity value for normalization.
`--threshold`	`0.0`	ESS threshold.
`--engine_file_path`	`${ISAAC_ROS_WS}/src/isaac_ros_dnn_stereo_depth/resources/ess.engine`	ESS engine file path.

ROS Topics Subscribed

ROS Topic	Interface	Description
`/<camera>/left/image_compressed`	sensor_msgs/msg/CompressedImage	Left camera stream.
`/<camera>/left/camera_info`	sensor_msgs/msg/CameraInfo	Left camera intrinsics.
`/<camera>/right/image_compressed`	sensor_msgs/msg/CompressedImage	Right camera stream.
`/<camera>/right/camera_info`	sensor_msgs/msg/CameraInfo	Right camera intrinsics.

isaac_ros_data_replayer

Overview

Quickstart

Set Up Development Environment

Download Quickstart Assets

Build isaac_ros_data_replayer

Run Launch File

Replay a single camera

Visualize

Data Extraction

Troubleshooting

Isaac ROS Troubleshooting

Not all camera frames extracted from recording

API

Usage

ROS Launch Arguments

ROS Topics Published

Usage

ROS Run Arguments

ROS Topics Subscribed

`isaac_ros_data_replayer`

Build `isaac_ros_data_replayer`