Camera-based Perception with Isaac Perceptor on a Robot with the Nova Orin Developer Kit

To run and evaluate camera-based perception on your robot with the Nova Orin Developer Kit, please follow suggested mounting guidelines, and power on the Nova Orin Developer Kit.

Note

The Nova Orin Developer Kit has been unboxed and set up as expected by following Getting Started.

Mounting the Nova Orin Developer Kit on a Robot

In order to use the Nova Orin Developer Kit as the perception module on an existing robot, you shall mount it on your robot. It is essential that the developer kit is mounted rigidly and cannot move.

It is recommended to mount the Nova Orin Developer Kit such that the camera’s field of views are largely unobstructed. Make sure to orient the Nova Orin Developer Kit such that the front camera is pointing to the robot’s main direction of travel, and the bottom plate is parallel to the ground.

The developer kit can be mounted using either a tripod mount or the general purpose hole pattern on its bottom. Recommended mounting hole positions can be found in the Nova Orin DevKit Quick Start Guide from Segway.

To account for the mounting location when running Isaac Perceptor, a custom URDF file needs to be passed. Per default, the nova_developer_kit.urdf.xacro is used, which assumes that the Nova Orin Developer Kit is initialized on the ground plane. To define your own URDF file, you can use the following code snippet and store it at a suitable urdf_file_path:

<?xml version="1.0"?>
<robot name="please_specify_robot_name" xmlns:xacro="http://www.ros.org/wiki/xacro">
<xacro:arg name="robot_namespace" default="/"/>
<xacro:include filename="$(find nova_developer_kit_description)/urdf/nova_developer_kit_macro.urdf.xacro"/>

<link name="base_link"/>
<xacro:nova_developer_kit name="nova_developer_kit" parent="base_link" mount_is_root="true">
    <origin xyz="0.0 0.0 0.0" rpy="0.0 0.0 0.0"/>
</xacro:nova_developer_kit>
</robot>

<!-- Your Robot Description goes here. -->

The fields xyz and rpy parametrize the transformation between the Nova Orin Developer Kit and the provided parent link (e.g. base_link). Specifically, the field xyz represents a 3D translation in meters, and the field rpy represents a rotation as fixed-axis Euler angles in radians: roll, pitch and yaw, respectively.

The field mount_is_root must be set to true if the transformation is provided with respect to the Nova Orin Developer Kit mount point (center screw of the lower plate), or false if the transformation refers to the Nova Orin Developer Kit main link (center-front of the module).

For example, if you are mounting the Nova Orin Developer Kit such that its center-front point is at a height of Z = 0.4 m above the robot origin and X = -0.1 m behind it, the root link origin should be adjusted accordingly:

...
<xacro:nova_developer_kit name="nova_developer_kit" parent="base_link" mount_is_root="false">
    <origin xyz="-0.1 0.0 0.4" rpy="0.0 0.0 0.0"/>
...

Your custom URDF file can then be passed with the urdf_override_file argument to Isaac Perceptor:

ros2 launch nova_developer_kit_bringup perceptor.launch.py \
    urdf_override_file:=<"urdf_file_path">

Powering On the Nova Orin Developer Kit

Use the power splitter cable to distribute power to the barrel jack ports found on the Jetson AGX Orin and GMSL Camera Board. For the input side of the splitter cable, provide a 12V (90W capable) power source. The male barrel jack input has an outer diameter of 5.5mm and inner diameter of 2.5mm. The system should turn on when power is applied. Alternatively, power on the Nova Orin Developer Kit by pressing the power button (the left-most button) in the AGX Orin.

Tutorials

You are recommended to go through following tutorials to running and evaluating Isaac Perceptor on your robot equipped with the Nova Orin Developer Kit.

Running Isaac Perceptor on a Robot

Follow the Tutorial: Running Camera-based 3D Perception with Isaac Perceptor on how to launch the application, how to visualize, and evaluate the results.

Running Isaac Perceptor on Recorded Data

See the Tutorial: Recording and Playing Back Data for Isaac Perceptor on how to record data on your robot equipped with Nova Orin Developer Kit, and how to run Isaac Perceptor on recordings.