============== |package_name| ============== :ir_github:` ` Quickstart ---------- Set Up Development Environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. include:: /_snippets/set_up_dev_env.rst Download Quickstart Assets ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Download quickstart data from NGC: :ir_assets:` ` Build |package_name| ~~~~~~~~~~~~~~~~~~~~ .. tabs:: .. tab:: Binary Package 1. Launch the Docker container using the ``run_dev.sh`` script: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 2. Install the prebuilt Debian package: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-detectnet ros-humble-isaac-ros-dnn-image-encoder ros-humble-isaac-ros-triton .. tab:: Build from Source 1. Clone this repository under ``${ISAAC_ROS_WS}/src``: .. code:: bash cd ${ISAAC_ROS_WS}/src && \ git clone :ir_clone:`` 2. Launch the Docker container using the ``run_dev.sh`` script: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 3. Use ``rosdep`` to install the package's dependencies: :ir_apt: .. code:: bash rosdep update && rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_object_detection/isaac_ros_detectnet --ignore-src -y 4. Build the package from source: .. code:: bash cd ${ISAAC_ROS_WS} && \ colcon build --symlink-install --packages-up-to isaac_ros_detectnet --base-paths ${ISAAC_ROS_WS}/src/isaac_ros_object_detection/isaac_ros_detectnet 5. Source the ROS workspace: .. note:: Make sure to repeat this step in **every** terminal created inside the Docker container. Since this package was built from source, the enclosing workspace must be sourced for ROS to be able to find the package's contents. .. code:: bash source install/setup.bash Run Launch File ~~~~~~~~~~~~~~~ .. tabs:: .. tab:: Rosbag #. Continuing inside the Docker container, run the quickstart setup script which will download the `PeopleNet Model `__ from NVIDIA GPU Cloud(NGC) .. code:: bash ros2 run isaac_ros_detectnet setup_model.sh --config-file quickstart_config.pbtxt #. Continuing inside the Docker container, install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples #. Run the following launch file to spin up a demo of this package using the quickstart rosbag: .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=detectnet interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_detectnet/quickstart_interface_specs.json #. Open a new terminal inside the Docker container: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh #. Run the rosbag file to simulate an image stream: .. code:: bash ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_detectnet/rosbags/detectnet_rosbag --remap image:=image_rect camera_info:=camera_info_rect .. tab:: RealSense Camera #. Continuing inside the Docker container, run the quickstart setup script which will download the `PeopleNet Model `__ from NVIDIA GPU Cloud(NGC) .. code:: bash ros2 run isaac_ros_detectnet setup_model.sh --config-file peoplenet_config.pbtxt #. Ensure that you have already set up your RealSense camera using the :doc:`RealSense setup tutorial `. If you have not, please set up the sensor and then restart this quickstart from the beginning. #. Continuing inside the Docker container, install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-realsense #. Run the following launch file to spin up a demo of this package using a RealSense camera: .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect,detectnet .. tab:: Hawk Camera #. Continuing inside the Docker container, run the quickstart setup script which will download the `PeopleNet Model `__ from NVIDIA GPU Cloud(NGC) .. code:: bash ros2 run isaac_ros_detectnet setup_model.sh --config-file peoplenet_config.pbtxt #. Ensure that you have already set up your Hawk camera using the :doc:`Hawk setup tutorial `. If you have not, please set up the sensor and then restart this quickstart from the beginning. #. Continuing inside the Docker container, install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-argus-camera #. Run the following launch file to spin up a demo of this package using a Hawk camera: .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=argus_mono,rectify_mono,detectnet .. tab:: ZED Camera #. Ensure that you have already set up your ZED camera using :doc:`ZED setup tutorial `. #. Continuing inside the Docker container, install dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-stereo-image-proc ros-humble-isaac-ros-zed #. Continuing inside the Docker container, run the quickstart setup script which will download the `PeopleNet Model `__ from NVIDIA GPU Cloud(NGC) .. note:: If you are using the `ZED X` series, change resolution and replace `zed2_config.pbtxt` with `zedx_config.pbtxt` in the below command. .. code:: bash ros2 run isaac_ros_detectnet setup_model.sh --config-file peoplenet_config.pbtxt #. Run the following launch file to spin up a demo of this package using a ZED Camera: .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py \ launch_fragments:=zed_mono_rect,detectnet \ interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_detectnet/zed2_quickstart_interface_specs.json .. note:: If you are using the `ZED X` series, replace `zed2_quickstart_interface_specs.json` with `zedx_quickstart_interface_specs.json` in the above command. Visualize Results ~~~~~~~~~~~~~~~~~ 1. Open a new terminal and run the ``isaac_ros_detectnet_visualizer``: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh .. code:: bash cd ${ISAAC_ROS_WS} && \ source install/setup.bash && \ ros2 run isaac_ros_detectnet isaac_ros_detectnet_visualizer.py --ros-args --remap image:=detectnet_encoder/resize/image 2. Open a new terminal and run ``rqt_image_view`` to visualize the output: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh .. code:: bash ros2 run rqt_image_view rqt_image_view /detectnet_processed_image 3. You should see an output as shown below: .. figure:: :ir_lfs:`` :alt: RQT showing detection of people :align: center Try More Examples ----------------- To continue your exploration, check out the following suggested examples: .. toctree:: :maxdepth: 1 Tutorial with Isaac Sim Tutorial with Custom Model This package only supports models based on the ``Detectnet_v2`` architecture. Some of the `supported DetectNet models `__ from NGC: ============================================================================================= ====================================================== Model Name Use Case ============================================================================================= ====================================================== `PeopleNet AMR `__ People counting with a mobile robot `PeopleNet `__ People counting, heatmap generation, social distancing `TrafficCamNet `__ Detect and track cars `DashCamNet `__ Identify objects from a moving object `FaceDetectIR `__ Detect faces in a dark environment with IR camera ============================================================================================= ====================================================== To learn how to use this models, click :doc:`here `. ROS 2 Graph Configuration ------------------------- To run the DetectNet object detection inference, the following ROS 2 nodes must be set up and running: .. figure:: :ir_lfs:`` :width: 800px :align: center 1. **Isaac ROS DNN Image encoder**: This takes an image message and converts it to a tensor (``TensorList`` :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList `) that can be processed by the network. 2. **Isaac ROS DNN Inference - Triton**: This executes the DetectNet network and takes, as input, the tensor from the DNN Image Encoder. .. note:: The :ir_repo:`Isaac ROS TensorRT ` package is not able to perform inference with DetectNet models at this time. The output is a TensorList message containing the encoded detections. Use the parameters ``model_name`` and ``model_repository_paths`` to point to the model folder and set the model name. The ``.plan`` file should be located at ``$model_repository_path/$model_name/1/model.plan`` 3. **Isaac ROS Detectnet Decoder**: This node takes the TensorList with encoded detections as input, and outputs ``Detection2DArray`` messages for each frame. See the following section for the parameters. Troubleshooting --------------- Isaac ROS Troubleshooting ~~~~~~~~~~~~~~~~~~~~~~~~~ For solutions to problems with Isaac ROS, see :doc:`troubleshooting `. Deep Learning Troubleshooting ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For solutions to problems with using DNN models, see :doc:`troubleshooting deeplearning `. API ---- Usage ~~~~~ .. code:: bash ros2 launch isaac_ros_detectnet isaac_ros_detectnet.launch.py label_list:= enable_confidence_threshold:= enable_bbox_area_threshold:= enable_dbscan_clustering:= confidence_threshold:= min_bbox_area:= dbscan_confidence_threshold:= dbscan_eps:= dbscan_min_boxes:= dbscan_enable_athr_filter:= dbscan_threshold_athr:= dbscan_clustering_algorithm:= bounding_box_scale:= bounding_box_offset:= ROS Parameters ~~~~~~~~~~~~~~ =============================== ============ ============================= =========================================================================================================================================================================================================================== ROS Parameter Type Default Description =============================== ============ ============================= =========================================================================================================================================================================================================================== ``label_list`` ``string[]`` ``{"person", "bag", "face"}`` The list of labels. These are loaded from ``labels.txt`` (downloaded with the model) ``confidence_threshold`` ``double`` ``0.35`` The min value of confidence used to threshold detections before clustering ``min_bbox_area`` ``double`` ``100`` The min value of bounding box area used to threshold detections before clustering ``dbscan_confidence_threshold`` ``double`` ``0.35`` Holds the epsilon to control merging of overlapping boxes. Refer to OpenCV ``groupRectangles`` and DBSCAN documentation for more information on epsilon. ``dbscan_eps`` ``double`` ``0.7`` Holds the epsilon to control merging of overlapping boxes. Refer to OpenCV ``groupRectangles`` and DBSCAN documentation for more information on epsilon. ``dbscan_min_boxes`` ``int`` ``1`` The minimum number of boxes to return. ``dbscan_enable_athr_filter`` ``int`` ``0`` Enables the area-to-hit ratio (ATHR) filter. The ATHR is calculated as: **ATHR = sqrt(clusterArea) / nObjectsInCluster.** ``dbscan_threshold_athr`` ``double`` ``0.0`` The ``area-to-hit`` ratio threshold. ``dbscan_clustering_algorithm`` ``int`` ``1`` The clustering algorithm selection. (``1``: Enables DBScan clustering, ``2``: Enables Hybrid clustering, resulting in more boxes that will need to be processed with NMS or other means of reducing overlapping detections. ``bounding_box_scale`` ``double`` ``35.0`` The scale parameter, which should match the training configuration. ``bounding_box_offset`` ``double`` ``0.0`` Bounding box offset for both X and Y dimensions. =============================== ============ ============================= =========================================================================================================================================================================================================================== ROS Topics Subscribed ~~~~~~~~~~~~~~~~~~~~~ ============== ==================================================================================================================================================================== =============================================================== ROS Topic Interface Description ============== ==================================================================================================================================================================== =============================================================== ``tensor_sub`` :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList ` The tensor that represents the inferred aligned bounding boxes. ============== ==================================================================================================================================================================== =============================================================== ROS Topics Published ~~~~~~~~~~~~~~~~~~~~ ======================== =============================================================================================================================== ================================================== ROS Topic Interface Description ======================== =============================================================================================================================== ================================================== ``detectnet/detections`` `vision_msgs/Detection2DArray `__ Aligned image bounding boxes with detection class. ======================== =============================================================================================================================== ================================================== .. |package_name| replace:: ``isaac_ros_detectnet``