NITROS

NVIDIA Isaac Transport for ROS (NITROS) is a technology to enable streaming through hardware-accelerated ROS graphs.

Motivation

ROS 2 Humble introduces new hardware-acceleration features, including type adaptation and type negotiation, that significantly increase performance for developers seeking to incorporate AI/machine learning and computer vision functionality into their ROS-based applications.

Type adaptation (REP-2007) is common for hardware accelerators, which require a different data format to deliver optimal performance. Type adaptation allows ROS nodes to work in a format better suited to the hardware. Processing graphs can eliminate memory copies between the CPU and the memory accelerator using the adapted type. Unnecessary memory copies consume CPU compute, waste power, and slow down performance, especially as the image size increases.

Type negotiation (REP-2009) allows different ROS nodes in a processing graph to advertise their supported types so that formats yielding ideal performance are chosen. The ROS framework performs this negotiation process and maintains compatibility with legacy nodes that don’t support negotiation.

Accelerating processing graphs using type adaptation and negotiation makes the hardware accelerator zero-copy possible. This reduces software/CPU overhead and unlocks the potential of the underlying hardware. As roboticists migrate to more powerful compute platforms like NVIDIA Jetson Orin, they can expect to realize more of the performance gains enabled by the hardware.

NITROS is NVIDIA’s implementation of type adaption and negotiation. ROS processing graphs made up of NITROS-based Isaac ROS hardware accelerated modules (a.k.a. GEMs or Isaac ROS nodes) can deliver promising performance and results.

System Assumptions

The design of NITROS makes the following assumptions of the ROS 2 applications:

To leverage the benefit of zero-copy in NITROS, all NITROS-accelerated nodes must run in the same process.
For a given topic in which type negotiation takes place, there can only be one negotiating publisher.
For a NITROS-accelerated node, received-frame IDs are assumed to be constant throughout the runtime.

NITROS-Accelerated Nodes

Most Isaac ROS GEMs have been updated to be NITROS-accelerated. The acceleration is in effect between NITROS-accelerated nodes when two or more of them are connected next to each other. In such a case, NITROS-accelerated nodes can discover each other through type negotiation and leverage type adaptation for data transmission automatically at runtime.

NITROS-accelerated nodes are also compatible with non-NITROS nodes: A NITROS-accelerated node can be used together with any existing, non-NITROS ROS 2 node, and it will function like a typical ROS 2 node.

CUDA with NITROS

For information about using CUDA with NITROS, please click here.

NITROS Data Types

NITROS supports transporting various common data types with zero-copy in its own NITROS types. Each NITROS type is one-to-one-mapped to a ROS message type, which ensures compatibility with existing tools, workflows, and codebases. A non-NITROS node supporting the corresponding ROS message types can publish data to or subscribe to data from a NITROS-accelerated node that supports the corresponding NITROS types.

Visit isaac_ros_nitros_type for a complete list of supported NITROS types. The following table lists a subset of commonly used types:

NITROS Interface	ROS Interface
NitrosImage	sensor_msgs/Image
NitrosCompressedImage	sensor_msgs/CompressedImage
NitrosCameraInfo	sensor_msgs/CameraInfo
NitrosTensorList	isaac_ros_tensor_list_interfaces/TensorList
NitrosDisparityImage	stereo_msgs/DisparityImage
NitrosPointCloud	sensor_msgs/PointCloud2
NitrosOccupancyGrid	nav_msgs/OccupancyGrid
NitrosOdometry	nav_msgs/Odometry
NitrosDetection2DArray	vision_msgs/Detection2D.msg
NitrosDetection3DArray	vision_msgs/Detection3D.msg
NitrosPoseArray	geometry_msgs/PoseArray
NitrosPoseCovStamped	geometry_msgs/PoseWithCovariance
NitrosTwist	geometry_msgs/Twist
NitrosImu	sensor_msgs/Imu
NitrosAprilTagDetectionArray	isaac_ros_apriltag_interfaces/AprilTagDetectionArray

NITROS-Accelerated Graphs

ROS 2 graphs built with NITROS-accelerated nodes yield promising performance. The following highlights three graphs that are created and tested fully with Isaac ROS NITROS-accelerated nodes. For more detailed performance outcomes, visit this page. To learn more about creating your own graphs with NITROS-accelerated nodes or adding NITROS-accelerated nodes in an existing non-NITROS graph, visit this page

AprilTag Detection Graph

The AprilTag detection graph uses the NVIDIA GPU-accelerated AprilTags library to detect AprilTags in images and publishes their poses, IDs, and additional metadata. Visit Isaac ROS AprilTag for more details.

graph LR; argus_node("ArgusMonoNode (Raw Image)") --> rectify_node("RectifyNode (Rectified Image)"); rectify_node --> apriltag_node("AprilTagNode (AprilTag Detection)");

Stereo Depth Graph

The stereo depth graph performs DNN-based stereo depth estimation via continuous disparity prediction. It produces a depth image or point cloud of the scene that can be used for robot navigation. Visit Isaac ROS DNN Stereo Depth for more details.

graph LR; argus_node("ArgusStereoNode (Raw Image)") --> left_rectify_node("RectifyNode (Rectified Image)"); argus_node --> right_rectify_node("RectifyNode (Rectified Image)"); left_rectify_node --> ess_node("ESSDisparityNode (DNN Inference)"); right_rectify_node --> ess_node; ess_node --> point_cloud_point("PointCloudNode (Point Cloud Output)");

Image Segmentation Graph

The image segmentation graph uses a deep learning U-Net model to generate an image mask segmenting out objects of interest. Visit Isaac ROS Image Segmentation <isaac_ros_image_segmentation> for more details.

graph LR; argus_node("ArgusMonoNode (Raw Image)") --> rectify_node("RectifyNode (Rectified Image)"); rectify_node --> encoder_node("DnnImageEncoderNode (DNN Pre-Processed Tensors)"); encoder_node --> triton_node("TritonNode (DNN Inference)"); triton_node --> unet_decoder_node("UNetDecoderNode (Segmentation Image)");

Repositories and Packages

The Isaac ROS implementations of this technology are available here:

Isaac ROS NITROS