Tutorial: Camera-based 3D Perception with Isaac Perceptor

This demo will enable you to use the Isaac Perceptor to perceive the robot’s environment - purely based on cameras. Isaac Perceptor uses Isaac ROS components, including Visual SLAM to localize the robot, ESS and nvblox to reconstruct 3D environments.

For this tutorial, it is assumed that you have successfully completed the teleop tutorial and the calibration tutorial.

Running the Application

1. SSH into the robot (instructions).

2. Make sure you have successfully connected the PS5 joystick to the robot (instructions).

3. Build/install the required packages and run the app:

Docker ImageBinary PackageBuild from Source

1. Pull the Docker image:

docker pull nvcr.io/nvidia/isaac/nova_carter_bringup:release_3.1-aarch64

2. Run the Docker image:

docker run --privileged --network host \
    -v /dev/*:/dev/* \
    -v /tmp/argus_socket:/tmp/argus_socket \
    -v /etc/nova:/etc/nova \
    nvcr.io/nvidia/isaac/nova_carter_bringup:release_3.1-aarch64 \
    ros2 launch nova_carter_bringup perceptor.launch.py

4. You are now able to remote control the robot with the gamepad. Follow the next steps to additionally visualize the sensor outputs in Foxglove.

Customizing Sensor Configurations

By default, as specified in the launch file perceptor.launch.py, 3 stereo cameras available on the Nova Carter (front, left, right) are used in Isaac Perceptor algorithms. You may use the stereo_camera_configuration launch argument to customize camera configurations when running Isaac Perceptor.

For example, to use only front stereo camera for 3D reconstruction and visual SLAM, you could run the following launch command:

ros2 launch nova_carter_bringup perceptor.launch.py \
    stereo_camera_configuration:=front_configuration

For example, to use only front stereo camera for 3D reconstruction, visual SLAM and people reconstruction, you could run the following launch command:

ros2 launch nova_carter_bringup perceptor.launch.py \
    stereo_camera_configuration:=front_people_configuration

For a detailed description of all available configurations refer to the Tutorial: Stereo Camera Configurations for Isaac Perceptor.

Visualizing the Outputs

1. Make sure you complete Visualization Setup. This is required to visualize the Isaac nvblox mesh in a recommended layout configuration.

2. Open the Foxglove studio on your remote machine.

   1. If you are not running a configuration with people reconstruction, open the nova_carter_perceptor.json layout file downloaded in the previous step.

   2. If you are running a configuration with people reconstruction, open the nova_carter_perceptor_with_people.json layout file downloaded in the previous step.

3. In Foxglove, it shows a visualization of the Nova Carter robot and Isaac nvblox mesh visualization of surrounding environments. You should expect a visualization similar as below. In the mesh, it shows the reconstructed colored voxels, the computed distance map from Isaac nvblox outputs. The colored voxels are uniformly reconstructed with a resolution of 5cm. The rainbow color spectrum reflects the proximity of each region to nearest obstacles. Regions closer to obstacle surfaces are marked in warmer colors (red, orange), while regions further away from obstacle surfaces are marked in cooler colors (blue, violet).

4. If running a configuration with people reconstruction, in addition, you can see highlighted red voxels shown in the reconstructed mesh. It visualizes people perceived in the field of view of one or more cameras. In the default nova_carter_perceptor_with_people.json layout file, only /nvblox_human_node/dynamic_occupancy_layer is visualized and shown as the red voxels.

Note

For a better visualization experience, some topics requiring a large bandwidth are not available to Foxglove studio. You can set use_foxglove_whitelist:=False as additional argument when running the app. Most likely the image stream will be fairly choppy given its large bandwidth. To learn more about topics published by Isaac nvblox, you can refer to nvblox ROS messages.. For topics published by Isaac visual SLAM, you can refer to cuvslam ROS messages..

Evaluating Isaac Perceptor

Follow these instructions to assert that Isaac Perceptor is performing as expected.

1. Isaac nvblox continuously measures computation efficiency metrics and reports them in the terminal. You should expect to see metrics similar to the following:

   1. nvblox Rates reports how the frequencies of specific events happening in nvblox. For the 3-camera configuration, you are expected to see ros/depth_image_callback at 60Hz, and ros/color_image_callback at 90Hz. For the 1-camera configuration, you are expected to see ros/depth_image_callback at 30Hz, and ros/color_image_callback at 30Hz.

   nvblox Rates (in Hz)
   namespace/tag - NumSamples (Window Length) - Mean
   -----------
   ros/color                   100              13.2
   ros/depth                   100              60.4
   ros/depth_image_callback    100              60.8
   ros/color_image_callback    100              90.8
   ros/update_esdf             100              7.2
   mapper/stream_mesh          100              4.5
   ros/tick                    100              16.7
   -----------
   

   2. nvblox Delays reports the average compute time a node takes in second based on a certain number of samples. You are expected to see ros/esdf_integration at 100 ~ 200 ms.

   nvblox Delays
   namespace/tag - NumSamples (Window Length) - Mean Delay (seconds)
   -----------
   ros/esdf_integration        100              0.174
   ros/depth_image_integration 100              0.152
   ros/color_image_integration 100              0.129
   ros/depth_image_callback    100              0.106
   ros/color_image_callback    100              0.068
   -----------
   

2. To further evaluate the quality of Isaac Perceptor, you can perform the following tests.

   1. Choose objects above 10cm with various heights, and place them in front of any camera specified in the stereo_camera_configuration launch argument. Vary the distance to the camera (e.g. 1m, 3m, 5m, 7m). You should expect to see the object in the distance map visualization. Additionally, you may also use the Measure Tool in the top panel to measure the distance between the object surface and the camera center. An example evaluation performed using the Nova Orin Developer Kit can be found in Evaluating Isaac Perceptor on the Nova Orin Developer Kit.

   2. Ensure Nova Carter is stationary. You are expected to see no drift from the Nova Carter robot visualization with reference to the odometry frame. You are advised to follow cuVSLAM to understand how Visual SLAM works.

Running Isaac Perceptor on Recorded Data

Apart from running camera-based perception on Nova Carter robot, see the Tutorial: Recording and Playing Back Data for Isaac Perceptor on how to record data on the Nova Carter robot and running Isaac Perceptor on the recorded data.