==============
|package_name|
==============

:ir_github:`<isaac_ros_object_detection> <isaac_ros_rtdetr>`

.. figure:: :ir_lfs:`<resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_rtdetr/syntheticadetr_1.0_grasp_objects.gif>`
   :align: center
   :width: 400px

   Stable object detections using SyntheticaDETR in a difficult scene with camera motion blur, round objects with few features, reflective object material, and light reflections

Quickstart
----------

Set Up Development Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. include:: /_snippets/set_up_dev_env.rst

Download Quickstart Assets
~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Download quickstart data from NGC:

   :ir_assets:`<isaac_ros_rtdetr> <quickstart.tar.gz>`

2. Download a pre-trained :ir_ngc:`SyntheticaDETR <teams/isaac/models/synthetica_detr>` model to use in the quickstart:

   .. code:: bash

      mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr && \
      cd ${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr && \
         wget 'https://api.ngc.nvidia.com/v2/models/nvidia/isaac/synthetica_detr/versions/1.0.0_onnx/files/sdetr_grasp.onnx'

   .. note::

      This quickstart uses the NVIDIA-produced ``sdetr_grasp`` SyntheticaDETR model, but Isaac ROS RT-DETR is compatible with all RT-DETR architecture models.
      For more about the differences between SyntheticaDETR and RT-DETR, see :doc:`here </concepts/object_detection/rtdetr/index>`.

Build |package_name|
~~~~~~~~~~~~~~~~~~~~

.. tabs::

   .. tab:: Binary Package

      1. Launch the Docker container using the ``run_dev.sh`` script:

         .. code:: bash

            cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
            ./scripts/run_dev.sh

      2. Install the prebuilt Debian package:

         :ir_apt:

         .. code:: bash

            sudo apt-get install -y ros-humble-isaac-ros-rtdetr
   
   .. tab:: Build from Source

      1. Clone this repository under ``${ISAAC_ROS_WS}/src``:

         .. code:: bash

            cd ${ISAAC_ROS_WS}/src && \
               git clone :ir_clone:`<isaac_ros_object_detection>`
     
      2. Launch the Docker container using the ``run_dev.sh`` script:

         .. code:: bash

            cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
            ./scripts/run_dev.sh

      3. Use ``rosdep`` to install the package's dependencies:

         :ir_apt:

         .. code:: bash

            rosdep update && rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_object_detection/isaac_ros_rtdetr --ignore-src -y

      4. Build the package from source:

         .. code:: bash

            cd ${ISAAC_ROS_WS} && \
               colcon build --symlink-install --packages-up-to isaac_ros_rtdetr --base-paths ${ISAAC_ROS_WS}/src/isaac_ros_object_detection/isaac_ros_rtdetr

      5. Source the ROS workspace:

         .. note::

            Make sure to repeat this step in **every** terminal created inside the Docker container.
            
            Since this package was built from source, the enclosing workspace must be sourced for ROS to be able to find the package's contents.

         .. code:: bash

            source install/setup.bash

Run Launch File
~~~~~~~~~~~~~~~

1. Continuing inside the Docker container, convert the model (``.onnx``) to a TensorRT engine plan:

   .. code:: bash

      /usr/src/tensorrt/bin/trtexec --onnx=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.onnx --saveEngine=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan

   .. note::

      The model conversion time varies across different platforms. On Jetson AGX Orin, for example, the engine conversion process can take up to 10-15 minutes to complete.

.. tabs::

   .. tab:: Rosbag

      #. Continuing inside the Docker container, install the following dependencies:

         :ir_apt:

         .. code:: bash

            sudo apt-get install -y ros-humble-isaac-ros-examples

      #. Run the following launch file to spin up a demo of this package using the quickstart rosbag:

         .. code:: bash

            ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=rtdetr interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_rtdetr/quickstart_interface_specs.json engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan

      #. Open a **second** terminal inside the Docker container:

         .. code:: bash

            cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
            ./scripts/run_dev.sh

      #. Run the rosbag file to simulate an image stream:

         .. code:: bash

            ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_rtdetr/quickstart.bag

   .. tab:: RealSense Camera

      #. Ensure that you have already set up your RealSense camera using the :doc:`RealSense setup tutorial </getting_started/hardware_setup/sensors/realsense_setup>`. If you have not, please set up the sensor and then restart this quickstart from the beginning.

      #. Continuing inside the Docker container, install the following dependencies:

         :ir_apt:

         .. code:: bash

            sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-realsense

      #. Run the following launch file to spin up a demo of this package using a RealSense camera:

         .. code:: bash

            ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect,rtdetr engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan
         
      .. include:: ../_snippets/sdetr_objects.rst

   .. tab:: Hawk Camera
         
      #. Ensure that you have already set up your Hawk camera using the :doc:`Hawk setup tutorial </getting_started/hardware_setup/sensors/hawk_setup>`. If you have not, please set up the sensor and then restart this quickstart from the beginning.

      #. Continuing inside the Docker container, install the following dependencies:

         :ir_apt:

         .. code:: bash

            sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-argus-camera

      #. Run the following launch file to spin up a demo of this package using a Hawk camera:

         .. code:: bash

            ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=argus_mono,rectify_mono,rtdetr  engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan

      .. include:: ../_snippets/sdetr_objects.rst

   .. tab:: ZED Camera

      #. Ensure that you have already set up your ZED camera using :doc:`ZED setup tutorial </getting_started/hardware_setup/sensors/zed_setup>`.

      #. Continuing inside the Docker container, install dependencies:

         :ir_apt:

         .. code:: bash
            
            sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-image-proc ros-humble-isaac-ros-zed
      
      #. Run the following launch file to spin up a demo of this package using a ZED Camera:

         .. code:: bash
            
            ros2 launch isaac_ros_examples isaac_ros_examples.launch.py \
            launch_fragments:=zed_mono_rect,rtdetr engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan \
            interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_rtdetr/zed2_quickstart_interface_specs.json
         
         .. note::
            
            If you are using the `ZED X` series, replace  `zed2_quickstart_interface_specs.json` with `zedx_quickstart_interface_specs.json` in the above command.

      .. include:: ../_snippets/sdetr_objects.rst

Visualize Results
^^^^^^^^^^^^^^^^^

1. Open a **new** terminal inside the Docker container:

   .. code:: bash

      cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
         ./scripts/run_dev.sh

2. Run the RT-DETR visualization script:

   .. code:: bash

      ros2 run isaac_ros_rtdetr isaac_ros_rtdetr_visualizer.py 

3. Open **another** terminal inside the Docker container:

   .. code:: bash

      cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
         ./scripts/run_dev.sh
         
4. Visualize and validate the output of the package with ``rqt_image_view``:
   
   .. code:: bash

      ros2 run rqt_image_view rqt_image_view /rtdetr_processed_image
      
   After **about 1 minute**, your output should look like this:

   .. figure:: :ir_lfs:`<resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_rtdetr/rqt_visualizer.png>`
      :alt: RQT showing detection of grocery objects
      :align: center

Try More Examples
-----------------

To continue your exploration, check out the following suggested examples:

.. toctree::
   :maxdepth: 1

   Tutorial with Isaac Sim </concepts/object_detection/rtdetr/tutorial_isaac_sim.rst>

This package only supports models based on the ``RT-DETR`` architecture. Some of the supported RT-DETR models from NGC:

.. list-table::
   :header-rows: 1

   * - Model Name
     - Use Case

   * - :ir_ngc:`sdetr_grasp <teams/isaac/models/synthetica_detr>`
     - Model trained on 100% synthetic data for object classes that can be grasped by a standard robot arm

   * - :ir_ngc:`sdetr_amr <teams/isaac/models/synthetica_detr>`
     - Model trained on 100% synthetic data for object classes that are relevant to the operation of an Autonomous Mobile Robot

To learn how to use these models, click :doc:`here </concepts/dnn_inference/model_preparation>`.

Troubleshooting
---------------

Isaac ROS Troubleshooting
~~~~~~~~~~~~~~~~~~~~~~~~~

For solutions to problems with Isaac ROS, see :doc:`troubleshooting </troubleshooting/index>`.

Deep Learning Troubleshooting
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
For solutions to problems with using DNN models, see :doc:`troubleshooting deeplearning </troubleshooting/deep_learning>`.

API
----

Usage
~~~~~

.. code:: bash

   ros2 launch isaac_ros_rtdetr isaac_ros_rtdetr.launch.py model_file_path:=<path to .onnx> engine_file_path:=<path to .plan> input_tensor_names:=<input tensor names> input_binding_names:=<input binding names> output_tensor_names:=<output tensor names> output_binding_names:=<output binding names> verbose:=<TensorRT verbosity> force_engine_update:=<force TensorRT update>

RtDetrPreprocessorNode
~~~~~~~~~~~~~~~~~~~~~~

ROS Parameters
^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - ROS Parameter
     - Type
     - Default
     - Description

   * - ``input_image_tensor_name``
     - ``string``
     - ``input_tensor``
     - The name of the encoded image tensor binding in the input tensor list.

   * - ``output_image_tensor_name``
     - ``string``
     - ``images``
     - The name of the encoded image tensor binding in the output tensor list.

   * - ``output_size_tensor_name``
     - ``string``
     - ``orig_target_sizes``
     - The name of the target image size tensor binding in the output tensor list.

   * - ``image_height``
     - ``int``
     - ``480``
     - The height of the original image, for resizing the final bounding box to match the original dimensions.
   
   * - ``image_width``
     - ``int``
     - ``640``
     - The width of the original image, for resizing the final bounding box to match the original dimensions.

ROS Topics Subscribed
^^^^^^^^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - ROS Topic
     - Interface
     - Description

   * - ``encoded_tensor``
     - :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList <isaac_ros_common> <isaac_ros_tensor_list_interfaces/msg/TensorList.msg>` 
     - The tensor that contains the encoded image data.

ROS Topics Published
~~~~~~~~~~~~~~~~~~~~

.. list-table::
   :header-rows: 1

   * - ROS Topic
     - Interface
     - Description

   * - ``tensor_pub``
     - :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList <isaac_ros_common> <isaac_ros_tensor_list_interfaces/msg/TensorList.msg>` 
     - Tensor list containing encoded image data and image size tensors.

RtDetrDecoderNode
~~~~~~~~~~~~~~~~~

ROS Parameters
^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - ROS Parameter
     - Type
     - Default
     - Description

   * - ``labels_tensor_name``
     - ``string``
     - ``labels``
     - The name of the labels tensor binding in the input tensor list.

   * - ``boxes_tensor_name``
     - ``string``
     - ``boxes``
     - The name of the boxes tensor binding in the input tensor list.

   * - ``scores_tensor_name``
     - ``string``
     - ``scores``
     - The name of the scores tensor binding in the input tensor list.

   * - ``confidence_threshold``
     - ``double``
     - ``0.9``
     - The minimum score required for a particular bounding box to be published.

ROS Topics Subscribed
^^^^^^^^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - ROS Topic
     - Interface
     - Description

   * - ``tensor_sub``
     - :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList <isaac_ros_common> <isaac_ros_tensor_list_interfaces/msg/TensorList.msg>` 
     - The tensor that represents the inferred aligned bounding boxes, labels, and scores.

ROS Topics Published
~~~~~~~~~~~~~~~~~~~~

.. list-table::
   :header-rows: 1

   * - ROS Topic
     - Interface
     - Description

   * - ``detections_output``
     - `vision_msgs/Detection2DArray <https://github.com/ros-perception/vision_msgs/blob/ros2/vision_msgs/msg/Detection2DArray.msg>`__ 
     - Aligned image bounding boxes with detection class

.. |package_name| replace:: ``isaac_ros_rtdetr``