Isaac ROS Object Detection

https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_object_detection_example.png/ https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_object_detection_example_bbox.png/

Overview

Isaac ROS Object Detection contains ROS 2 packages to perform object detection. isaac_ros_rtdetr, isaac_ros_detectnet, and isaac_ros_yolov8 each provide a method for spatial classification using bounding boxes with an input image. Classification is performed by a GPU-accelerated model of the appropriate architecture:

The output prediction can be used by perception functions to understand the presence and spatial location of an object in an image.

https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_object_detection_nodegraph.png/

Each Isaac ROS Object Detection package is used in a graph of nodes to provide a bounding box detection array with object classes from an input image. A trained model of the appropriate architecture is required to produce the detection array.

Input images may need to be cropped and resized to maintain the aspect ratio and match the input resolution of the specific object detection model; image resolution may be reduced to improve DNN inference performance, which typically scales directly with the number of pixels in the image. isaac_ros_dnn_image_encoder provides DNN encoder utilities to process the input image into Tensors for the object detection models. Prediction results are decoded in model-specific ways, often involving clustering and thresholding to group multiple detections on the same object and reduce spurious detections. Output is provided as a detection array with object classes.

DNNs have a minimum number of pixels that need to be visible on the object to provide a classification prediction. If a person cannot see the object in the image, it’s unlikely the DNN will. Reducing input resolution to reduce compute may reduce what is detected in the image. For example, a 1920x1080 image containing a distant person occupying 1k pixels (64x16) would have 0.25K pixels (32x8) when downscaled by 1/2 in both X and Y. The DNN may detect the person with the original input image, which provides 1K pixels for the person, and fail to detect the same person in the downscaled resolution, which only provides 0.25K pixels for the person.

https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_object_detection_example_bboxseg.png/

Object detection classifies a rectangle of pixels as containing an object, whereas image segmentation provides more information and uses more compute to produce a classification per pixel. Object detection is used to know if, and where in a 2D image, the object exists. If a 3D spacial understanding or size of an object in pixels is required, use image segmentation.

Quickstarts

Isaac ROS NITROS Acceleration

This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.

DNN Models

To perform DNN inferencing a DNN model is required. NGC provides SyntheticaDETR and DetectNet pre-trained models for use in your robotics application. Using TAO these pre-trained models can be fine-tuned for your application.

Click here for more information on how to use NGC models.

Packages

Supported Platforms

This package is designed and tested to be compatible with ROS 2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.

Note

Versions of ROS 2 other than Humble are not supported. This package depends on specific ROS 2 implementation features that were introduced beginning with the Humble release. ROS 2 versions after Humble have not yet been tested.

Platform

Hardware

Software

Notes

Jetson

Jetson Orin

JetPack 6.0

For best performance, ensure that power settings are configured appropriately.


Jetson Orin Nano 4GB may not have enough memory to run many of the Isaac ROS packages and is not recommended.

x86_64

Ampere or higher NVIDIA GPU Architecture with 8 GB RAM or higher

Ubuntu 22.04+

CUDA 12.2+

Docker

To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.

Note

All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.

Customize your Dev Environment

To customize your development environment, reference this guide.

Updates

Date

Changes

2024-09-26

Update for ZED compatibility

2024-05-30

Added RT-DETR object detection package

2023-10-18

Adding NITROS YOLOv8 decoder

2023-05-25

Performance improvements

2023-04-05

Source available GXF extensions

2022-10-19

Updated OSS licensing

2022-08-31

Update to use NITROS for improved performance and to be compatible with JetPack 5.0.2

2022-06-30

Support for ROS 2 Humble and miscellaneous bug fixes

2022-03-21

Initial release