Isaac ROS Object Detection

Overview

Isaac ROS Object Detection contains an ROS 2 package to perform object detection. isaac_ros_detectnet provides a method for spatial classification using bounding boxes with an input image. Classification is performed by a GPU-accelerated DetectNet model. The output prediction can be used by perception functions to understand the presence and spatial location of an object in an image.

isaac_ros_detectnet is used in a graph of nodes to provide a bounding box detection array with object classes from an input image. A DetectNet model is required to produce the detection array. Input images may need to be cropped and resized to maintain the aspect ratio and match the input resolution of DetectNet; image resolution may be reduced to improve DNN inference performance, which typically scales directly with the number of pixels in the image. isaac_ros_dnn_image_encoder provides a DNN encoder to process the input image into Tensors for the DetectNet model. Prediction results are clustered in the DNN decoder to group multiple detections on the same object. Output is provided as a detection array with object classes.

DNNs have a minimum number of pixels that need to be visible on the object to provide a classification prediction. If a person cannot see the object in the image, it’s unlikely the DNN will. Reducing input resolution to reduce compute may reduce what is detected in the image. For example, a 1920x1080 image containing a distant person occupying 1k pixels (64x16) would have 0.25K pixels (32x8) when downscaled by 1/2 in both X and Y. The DNN may detect the person with the original input image, which provides 1K pixels for the person, and fail to detect the same person in the downscaled resolution, which only provides 0.25K pixels for the person.

Note

DetectNet is similar to other popular object detection models such as YOLOV3, FasterRCNN, and SSD, while being efficient at detecting multiple object classes in large images.

Object detection classifies a rectangle of pixels as containing an object, whereas image segmentation provides more information and uses more compute to produce a classification per pixel. Object detection is used to know if, and where in a 2D image, the object exists. If a 3D spacial understanding or size of an object in pixels is required, use image segmentation.

Quickstarts

• Isaac ROS Detectnet

• Isaac ROS YOLOv8

Isaac ROS NITROS Acceleration

This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.

DNN Models

To perform DNN inferencing a DNN model is required. NGC provides DetectNet pre-trained models for use in your robotics application. Using TAO these pre-trained models can be fine-tuned for your application.

Click here for more information on how to use NGC models.

Packages

• isaac_ros_detectnet
  • Quickstart
  • Try More Examples
  • ROS 2 Graph Configuration
  • Troubleshooting
  • API
• isaac_ros_yolov8
  • Quickstart
  • API

Supported Platforms

This package is designed and tested to be compatible with ROS 2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.

Note

Versions of ROS 2 earlier than Humble are not supported. This package depends on specific ROS 2 implementation features that were only introduced beginning with the Humble release.

Platform	Hardware	Software	Notes
Jetson	Jetson Orin Jetson Xavier	JetPack 5.1.2	For best performance, ensure that power settings are configured appropriately.
x86_64	NVIDIA GPU	Ubuntu 20.04+ CUDA 11.8+

Docker

To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.

Note

All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.

Customize your Dev Environment

To customize your development environment, reference this guide.

Updates

Date	Changes
2023-10-18	Adding NITROS YOLOv8 decoder
2023-05-25	Performance improvements
2023-04-05	Source available GXF extensions
2022-10-19	Updated OSS licensing
2022-08-31	Update to use NITROS for improved performance and to be compatible with JetPack 5.0.2
2022-06-30	Support for ROS 2 Humble and miscellaneous bug fixes
2022-03-21	Initial release