isaac_ros_data_validation

Source code on GitHub.

Overview

Tools to validate data from Nova. The isaac_ros_data_validation package primarily contains a utility to check ROSbags generated by isaac_ros_data_recorder for frame drops, desynchronization, and other issues.

Quickstart

Set Up Development Environment

  1. Set up your development environment by following the instructions in getting started.

  2. Clone isaac_ros_common under ${ISAAC_ROS_WS}/src.

    cd ${ISAAC_ROS_WS}/src && \
       git clone -b release-3.1 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common.git isaac_ros_common
    
  3. (Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

    Warning

    We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the Isaac ROS Dev container during installation, which will interrupt the quickstart process.

Download Quickstart Assets

  1. Download quickstart data from NGC:

    Make sure required libraries are installed.

    sudo apt-get install -y curl jq tar
    

    Then, run these commands to download the asset from NGC:

    NGC_ORG="nvidia"
    NGC_TEAM="isaac"
    PACKAGE_NAME="isaac_ros_data_validation"
    NGC_RESOURCE="isaac_ros_data_validation_assets"
    NGC_FILENAME="quickstart.tar.gz"
    MAJOR_VERSION=3
    MINOR_VERSION=1
    VERSION_REQ_URL="https://catalog.ngc.nvidia.com/api/resources/versions?orgName=$NGC_ORG&teamName=$NGC_TEAM&name=$NGC_RESOURCE&isPublic=true&pageNumber=0&pageSize=100&sortOrder=CREATED_DATE_DESC"
    AVAILABLE_VERSIONS=$(curl -s \
        -H "Accept: application/json" "$VERSION_REQ_URL")
    LATEST_VERSION_ID=$(echo $AVAILABLE_VERSIONS | jq -r "
        .recipeVersions[]
        | .versionId as \$v
        | \$v | select(test(\"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+$\"))
        | split(\".\") | {major: .[0]|tonumber, minor: .[1]|tonumber, patch: .[2]|tonumber}
        | select(.major == $MAJOR_VERSION and .minor <= $MINOR_VERSION)
        | \$v
        " | sort -V | tail -n 1
    )
    if [ -z "$LATEST_VERSION_ID" ]; then
        echo "No corresponding version found for Isaac ROS $MAJOR_VERSION.$MINOR_VERSION"
        echo "Found versions:"
        echo $AVAILABLE_VERSIONS | jq -r '.recipeVersions[].versionId'
    else
        mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets && \
        FILE_REQ_URL="https://api.ngc.nvidia.com/v2/resources/$NGC_ORG/$NGC_TEAM/$NGC_RESOURCE/\
    versions/$LATEST_VERSION_ID/files/$NGC_FILENAME" && \
        curl -LO --request GET "${FILE_REQ_URL}" && \
        tar -xf ${NGC_FILENAME} -C ${ISAAC_ROS_WS}/isaac_ros_assets && \
        rm ${NGC_FILENAME}
    fi
    

Build isaac_ros_data_validation

  1. Launch the Docker container using the run_dev.sh script:

    cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
    ./scripts/run_dev.sh
    
  2. Install the prebuilt Debian package:

    sudo apt-get install -y ros-humble-isaac-ros-data-validation
    

Run Launch File

  1. Continuing inside the Docker container, run data validation:

    python -m isaac_ros_data_validation.summarize_bag ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_data_validation/quickstart
    

    This will generate the report:

    ======================================================== example.mcap ========================================================
    Camera Drop Q Score: 100.0
    Camera Bucket Q Score: 100.0
    Intra Camera Sync Q score: 100.0
    
    Camera Drop Table                          [................................................................]
    /back_stereo_camera/left                   [................................................................]
    /back_stereo_camera/right                  [................................................................]
    /front_stereo_camera/left                  [................................................................]
    /front_stereo_camera/right                 [................................................................]
    /left_stereo_camera/left                   [................................................................]
    /left_stereo_camera/right                  [................................................................]
    /right_stereo_camera/left                  [................................................................]
    /right_stereo_camera/right                 [................................................................]
    
    Intra Camera Sync Table                    [................................................................]
    /back_stereo_camera/camera_info            [................................................................]
    /back_stereo_camera/image_compressed       [................................................................]
    /front_stereo_camera/camera_info           [................................................................]
    /front_stereo_camera/image_compressed      [................................................................]
    /left_stereo_camera/camera_info            [................................................................]
    /left_stereo_camera/image_compressed       [................................................................]
    /right_stereo_camera/camera_info           [................................................................]
    /right_stereo_camera/image_compressed      [................................................................]
    
    
    Topics:
        /left_stereo_camera/left/camera_info | frames_captured: 301
        /back_stereo_camera/left/camera_info | frames_captured: 300
        /left_stereo_camera/right/camera_info | frames_captured: 301
        /back_stereo_camera/right/camera_info | frames_captured: 300
        /chassis/odom | frames_captured: 400
        /right_stereo_camera/left/image_compressed | frames_captured: 300
        /front_stereo_imu/imu | frames_captured: 969
        /right_stereo_camera/right/image_compressed | frames_captured: 300
        /front_stereo_camera/left/image_compressed | frames_captured: 300
        /chassis/ticks | frames_captured: 400
        /chassis/imu | frames_captured: 400
        /front_stereo_camera/right/image_compressed | frames_captured: 300
        /back_stereo_camera/left/image_compressed | frames_captured: 300
        /left_stereo_camera/left/image_compressed | frames_captured: 300
        /back_stereo_camera/right/image_compressed | frames_captured: 300
        /left_stereo_camera/right/image_compressed | frames_captured: 300
        /right_stereo_camera/left/camera_info | frames_captured: 300
        /right_stereo_camera/right/camera_info | frames_captured: 300
        /front_stereo_camera/left/camera_info | frames_captured: 300
        /front_stereo_camera/right/camera_info | frames_captured: 300
        /chassis/battery_state | frames_captured: 10
    
    
    /back_stereo_camera/camera_info/sync:
        - Num Desyncs : 0
        - Percent Desynced: 0.0
        - Mean Difference : 0.0
        - Max Difference: [0.0]
        - Desync Table: ['................................................................']
    
    /back_stereo_camera/image_compressed/sync:
        - Num Desyncs : 0
        - Percent Desynced: 0.0
        - Mean Difference : 0.0
        - Max Difference: [0.0]
        - Desync Table: ['................................................................']
    
    /back_stereo_camera/left:
        - Percent Dropped: 0.0%
        - Number Dropped: 0
        - Mean Frequency: 30.00079422035675
        - Drop Table: [................................................................]
        - Frames Captured: 300
        - Total Jitter: 0.006412842809363813 ms
        - Jitter Excluding Drops: 0.006412842809363813 ms
        - Largest Frame Drop: 33.347072 ms
    
    /back_stereo_camera/right:
        - Percent Dropped: 0.0%
        - Number Dropped: 0
        - Mean Frequency: 30.00079422035675
        - Drop Table: [................................................................]
        - Frames Captured: 300
        - Total Jitter: 0.006412842809363813 ms
        - Jitter Excluding Drops: 0.006412842809363813 ms
        - Largest Frame Drop: 33.347072 ms
    
    /chassis/battery_state:
        - Percent Dropped/High-Jitter: (0.9,)%
        - Number Dropped/High-Jitter: 9
        - Mean Frequency: 0.9901003944077132
        - Drop / Jitter Table: [.......x......x......x......x......x......x......x......x......x]
        - Frames Captured: 10
        - Total Jitter: 976.6652543333333 ms
        - Jitter Excluding Drops: nan ms
    
    /chassis/imu:
        - Percent Dropped/High-Jitter: (0.0,)%
        - Number Dropped/High-Jitter: 0
        - Mean Frequency: 40.000823066559676
        - Drop / Jitter Table: [................................................................]
        - Frames Captured: 400
        - Total Jitter: 8.333847739348368 ms
        - Jitter Excluding Drops: 8.333847739348368 ms
    
    /chassis/odom:
        - Percent Dropped/High-Jitter: (0.0,)%
        - Number Dropped/High-Jitter: 0
        - Mean Frequency: 40.00078097564878
        - Drop / Jitter Table: [................................................................]
        - Frames Captured: 400
        - Total Jitter: 8.333821433583957 ms
        - Jitter Excluding Drops: 8.333821433583957 ms
    
    /front_stereo_camera/camera_info/sync:
        - Num Desyncs : 0
        - Percent Desynced: 0.0
        - Mean Difference : 0.0
        - Max Difference: [0.0]
        - Desync Table: ['................................................................']
    
    /front_stereo_camera/image_compressed/sync:
        - Num Desyncs : 0
        - Percent Desynced: 0.0
        - Mean Difference : 0.0
        - Max Difference: [0.0]
        - Desync Table: ['................................................................']
    
    /front_stereo_camera/left:
        - Percent Dropped: 0.0%
        - Number Dropped: 0
        - Mean Frequency: 30.000731030522118
        - Drop Table: [................................................................]
        - Frames Captured: 300
        - Total Jitter: 0.006407705685617994 ms
        - Jitter Excluding Drops: 0.006407705685617994 ms
        - Largest Frame Drop: 33.349888 ms
    
    /front_stereo_camera/right:
        - Percent Dropped: 0.0%
        - Number Dropped: 0
        - Mean Frequency: 30.000731030522118
        - Drop Table: [................................................................]
        - Frames Captured: 300
        - Total Jitter: 0.006407705685617994 ms
        - Jitter Excluding Drops: 0.006407705685617994 ms
        - Largest Frame Drop: 33.349888 ms
    
    /left_stereo_camera/camera_info/sync:
        - Num Desyncs : 0
        - Percent Desynced: 0.0
        - Mean Difference : 0.0
        - Max Difference: [0.0]
        - Desync Table: ['................................................................']
    
    /left_stereo_camera/image_compressed/sync:
        - Num Desyncs : 0
        - Percent Desynced: 0.0
        - Mean Difference : 0.0
        - Max Difference: [0.0]
        - Desync Table: ['................................................................']
    
    /left_stereo_camera/left:
        - Percent Dropped: 0.0%
        - Number Dropped: 0
        - Mean Frequency: 30.000721783251574
        - Drop Table: [................................................................]
        - Frames Captured: 300
        - Total Jitter: 0.006984775919731731 ms
        - Jitter Excluding Drops: 0.006984775919731731 ms
        - Largest Frame Drop: 33.350144 ms
    
    /left_stereo_camera/right:
        - Percent Dropped: 0.0%
        - Number Dropped: 0
        - Mean Frequency: 30.000721783251574
        - Drop Table: [................................................................]
        - Frames Captured: 300
        - Total Jitter: 0.006984775919731731 ms
        - Jitter Excluding Drops: 0.006984775919731731 ms
        - Largest Frame Drop: 33.350144 ms
    
    /right_stereo_camera/camera_info/sync:
        - Num Desyncs : 0
        - Percent Desynced: 0.0
        - Mean Difference : 0.0
        - Max Difference: [0.0]
        - Desync Table: ['................................................................']
    
    /right_stereo_camera/image_compressed/sync:
        - Num Desyncs : 0
        - Percent Desynced: 0.0
        - Mean Difference : 0.0
        - Max Difference: [0.0]
        - Desync Table: ['................................................................']
    
    /right_stereo_camera/left:
        - Percent Dropped: 0.0%
        - Number Dropped: 0
        - Mean Frequency: 30.00076031358312
        - Drop Table: [................................................................]
        - Frames Captured: 300
        - Total Jitter: 0.006439670011147545 ms
        - Jitter Excluding Drops: 0.006439670011147545 ms
        - Largest Frame Drop: 33.347072 ms
    
    /right_stereo_camera/right:
        - Percent Dropped: 0.0%
        - Number Dropped: 0
        - Mean Frequency: 30.00076031358312
        - Drop Table: [................................................................]
        - Frames Captured: 300
        - Total Jitter: 0.006439670011147545 ms
        - Jitter Excluding Drops: 0.006439670011147545 ms
        - Largest Frame Drop: 33.347072 ms
    

    The Drop Q Score is simply 1 - (frames dropped / total frames). The Bucket Q Score is what percent of equally spaced buckets on the timeline have no data loss. In the owl-4 example we only drop 20% of frames, but because the data loss occurs throughout the recording the bucket Q score is only 2%.

    The drop tables use an “X” table to show where in the recording a problem was found, in there is an issue with the batter_state not publishing at a constant rate.

    If you want to start doing further analysis you can look at the isaac_ros_data_validation/example directory. read_mcap_example.py is a good place to start or ess_disparities_example.py if you have a db3 file.

Troubleshooting

Isaac ROS Troubleshooting

For solutions to problems with Isaac ROS, see troubleshooting.

API

Usage

python -m isaac_ros_data_validation.summarize_bag input_file

Command Line Arguments

Command Line Argument

Default Value

Description

input_file

None

Path to rosbag.

-v/--verbosity

warning

Verbosity level. Can be dump, error, info, compact, or warning.