isaac_ros_yolov8

Source code available on GitHub.

Quickstart

Set Up Development Environment

1. Set up your development environment by following the instructions in getting started.

2. (Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

   Note

   We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the development environment during installation, which will interrupt the quickstart process.

Download Quickstart Assets

1. Download quickstart data from NGC:

   NVIDIA Internal:

   The NGC CLI must be set up once per system.

   1. Install the NGC CLI from here.

      Note

      Jetson Only: Do not edit the ~/.bash_profile file as suggested by the official NGC CLI instructions.

      Instead, add the NGC CLI to your PATH through ~/.bashrc:

      s="export PATH=\"\$PATH:$(pwd)/ngc-cli\""
      f="$HOME/.bashrc"; grep -qxF "$s" $f || echo "$s" | tee -a $f && source $f
      

   2. Generate an API key here.

   3. Run ngc config set:

      When prompted, enter the API key you generated earlier.

      ngc config set
      

   4. To confirm that the NGC CLI is set up correctly, run the following command:

      grep -E "^apikey\s*=\s*" ~/.ngc/config
      

      You should see an output with a valid API key:

      apikey = <your_api_key>
      

   Make sure required libraries are installed.

   sudo apt-get install -y curl jq tar
   

   Then, run these commands to download the asset from NGC:

   NGC_ORG="nvstaging"
   NGC_TEAM="isaac"
   PACKAGE_NAME="isaac_ros_yolov8"
   NGC_RESOURCE="isaac_ros_yolov8_assets"
   NGC_FILENAME="quickstart.tar.gz"
   MAJOR_VERSION=3
   MINOR_VERSION=3
   NGC_CONFIG_FILE="$HOME/.ngc/config"
   API_KEY=$(grep -E "^apikey\s*=\s*" $NGC_CONFIG_FILE | \
       awk -F "=" '{print $2}' | tr -d '[:space:]')
   TOKEN=$(curl -s -u "\$oauthtoken":"$API_KEY" -H 'Accept:application/json' \
       "https://authn.nvidia.com/token?service=ngc"\
   "&scope=group/ngc:$NGC_ORG&group/ngc:$NGC_ORG/$NGC_TEAM" | \
       jq -r '.token')
   VERSION_REQ_URL="https://api.ngc.nvidia.com/v2/org/$NGC_ORG/team/$NGC_TEAM/resources/$NGC_RESOURCE/versions"
   AVAILABLE_VERSIONS=$(curl -s -H "Authorization: Bearer ${TOKEN}" \
       -H "Accept: application/json" "$VERSION_REQ_URL")
   LATEST_VERSION_ID=$(echo $AVAILABLE_VERSIONS | jq -r "
       .recipeVersions[]
       | .versionId as \$v
       | \$v | select(test(\"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+$\"))
       | split(\".\") | {major: .[0]|tonumber, minor: .[1]|tonumber, patch: .[2]|tonumber}
       | select(.major == $MAJOR_VERSION and .minor <= $MINOR_VERSION)
       | \$v
       " | sort -V | tail -n 1
   )
   if [ -z "$LATEST_VERSION_ID" ]; then
       echo "No corresponding version found for Isaac ROS $MAJOR_VERSION.$MINOR_VERSION"
       echo "Found versions:"
       echo $AVAILABLE_VERSIONS | jq -r '.recipeVersions[].versionId'
   else
       mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets && \
       FILE_REQ_URL="https://api.ngc.nvidia.com/v2/org/$NGC_ORG/team/$NGC_TEAM/resources/\
   $NGC_RESOURCE/versions/$LATEST_VERSION_ID/files/$NGC_FILENAME" && \
       curl -LO --request GET "${FILE_REQ_URL}" \
           -H "Authorization: Bearer ${TOKEN}" \
           -H "Content-Type: application/json" && \
       tar -xf ${NGC_FILENAME} -C ${ISAAC_ROS_WS}/isaac_ros_assets && \
       rm ${NGC_FILENAME}
   fi
   

2. Download the model of your choice from Ultralytics YOLOv8. For this example, we use YOLOv8s.

   cd Downloads && \
      wget https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8s.pt
   

3. Convert the PyTorch model (.pt) to a general ONNX model (.onnx). Export to ONNX following instructions given below or here. Arguments can be specified for FP16 quantization during this step. This ONNX model is converted to a TensorRT engine file and used with the Isaac ROS TensorRT node for inference. You can use netron to visualize the ONNX model and note input and output names and dimensions.

   This can be done by first installing ultralytics and onnx via pip:

   pip3 install ultralytics
   pip3 install onnx
   

   Afterwards, convert the model from a .pt file to a .onnx model using ultralytics. This can be done by running:

   python3
   

   Then within python3, export the model:

   >> from ultralytics import YOLO
   >> model = YOLO('yolov8s.pt')
   >> model.export(format='onnx')
   

   Exit the interactive python shell and copy the generated .onnx model into the designated location for Isaac ROS (${ISAAC_ROS_WS}/isaac_ros_assets/models/yolov8):

   mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets/models/yolov8
   cp yolov8s.onnx ${ISAAC_ROS_WS}/isaac_ros_assets/models/yolov8
   

Build isaac_ros_yolov8

Binary PackageBuild from Source

1. Launch the Docker container using the run_dev.py script:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.py
   

2. Install the prebuilt Debian package:

   NVIDIA Internal: Run these commands to add the internal apt repository:

   sudo apt install curl -y
   k="/usr/share/keyrings/nvidia-isaac-ros.gpg"
   curl -fsSL https://isaac.download.nvidia.com/isaac-ros/repos.key | sudo gpg --dearmor | sudo tee -a $k > /dev/null
   f="/etc/apt/sources.list.d/nvidia-isaac-ros.list" && sudo touch $f
   s="deb [signed-by=$k] https://urm.nvidia.com/artifactory/sw-isaac-staging-debian-local jammy release-3.3"
   grep -qxF "$s" $f || echo "$s" | sudo tee -a $f
   
   pin_content=$'package: *\nPin: origin isaac.download.nvidia.com\nPin-Priority: 400'
   echo "$pin_content" | sudo tee /etc/apt/preferences.d/isaac-ros
   
   sudo apt-get update
   

   sudo apt-get install -y ros-humble-isaac-ros-yolov8 ros-humble-isaac-ros-dnn-image-encoder ros-humble-isaac-ros-tensor-rt
   

Run Launch File

1. Enter the Docker container in Jetson:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
      ./scripts/run_dev.py
   

RosbagRealSense CameraHawk CameraZED Camera

1. Continuing inside the Docker container, install the following dependencies:

   NVIDIA Internal: Run these commands to add the internal apt repository:

   sudo apt install curl -y
   k="/usr/share/keyrings/nvidia-isaac-ros.gpg"
   curl -fsSL https://isaac.download.nvidia.com/isaac-ros/repos.key | sudo gpg --dearmor | sudo tee -a $k > /dev/null
   f="/etc/apt/sources.list.d/nvidia-isaac-ros.list" && sudo touch $f
   s="deb [signed-by=$k] https://urm.nvidia.com/artifactory/sw-isaac-staging-debian-local jammy release-3.3"
   grep -qxF "$s" $f || echo "$s" | sudo tee -a $f
   
   pin_content=$'package: *\nPin: origin isaac.download.nvidia.com\nPin-Priority: 400'
   echo "$pin_content" | sudo tee /etc/apt/preferences.d/isaac-ros
   
   sudo apt-get update
   

   sudo apt-get install -y ros-humble-isaac-ros-examples
   

2. Run the following launch file to spin up a demo of this package using the quickstart rosbag:

   cd /workspaces/isaac_ros-dev && \
   ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=yolov8 interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_yolov8/quickstart_interface_specs.json \
      model_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/yolov8/yolov8s.onnx engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/yolov8/yolov8s.plan
   

3. Open a second terminal inside the Docker container:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.py
   

4. Run the rosbag file to simulate an image stream:

   ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_yolov8/quickstart.bag
   

Visualize Results

1. Open a new terminal inside the Docker container:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
      ./scripts/run_dev.py
   

2. Run the YOLOv8 visualization script:

   ros2 run isaac_ros_yolov8 isaac_ros_yolov8_visualizer.py
   

3. Open another terminal inside the Docker container:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
      ./scripts/run_dev.py
   

4. Visualize and validate the output of the package with rqt_image_view:

   ros2 run rqt_image_view rqt_image_view /yolov8_processed_image
   

   Your output should look like this:

   

Troubleshooting

Isaac ROS Troubleshooting

For solutions to problems with Isaac ROS, review here.

Deep Learning Troubleshooting

For solutions to problems with using DNN models, review here.

API

Usage

ros2 launch isaac_ros_yolov8 isaac_ros_yolov8_visualize.launch.py model_file_path:=<model_file_path> engine_file_path:=<engine_file_path> input_binding_names:=<input_binding_names> output_binding_names:=<output_binding_names> network_image_width:=<network_image_width> network_image_height:=<network_image_height> force_engine_update:=<force_engine_update> image_mean:=<image_mean> image_stddev:=<image_stddev> confidence_threshold:=<confidence_threshold> nms_threshold:=<nms_threshold>

Yolov8DecoderNode

ROS Parameters

ROS Parameter	Type	Default	Description
tensor_name	string	"output_tensor"	Name of the inferred output tensor published by the Managed NITROS Publisher. The decoder uses this name to get the output tensor.
confidence_threshold	float	0.25	Detection confidence threshold. Used to filter candidate detections during Non-Maximum Suppression (NMS).
nms_threshold	float	0.45	NMS IOU threshold.

ROS Topics Subscribed

ROS Topic	Interface	Description
tensor_sub	isaac_ros_tensor_list_interfaces/TensorList	Tensor list from the managed NITROS subscriber that represents the inferred aligned bounding boxes.

ROS Topics Published

ROS Topic	Interface	Description
detections_output	vision_msgs/Detection2DArray	Aligned image bounding boxes with detection class.