Tutorial for DOPE Inference with Triton ======================================= Overview --------- This tutorial walks you through a graph to estimate the 6DOF pose of a target object using :ir_repo:`DOPE ` using different backends. It uses input monocular images from a rosbag. The different backends show are: 1. PyTorch and ONNX 2. TensorRT Plan files with Triton 3. PyTorch model with Triton .. .. note:: The DOPE converter script only works on ``x86_64``, so the resultant ``onnx`` model following these steps must be copied to the Jetson. Tutorial Walkthrough -------------------- 1. Complete until ``Run Launch File`` of the quickstart :doc:`here `. 2. Make a directory called ``dope_ketchup`` inside ``${ISAAC_ROS_WS}/isaac_ros_assets/models``, which will serve as the model repository. This will be versioned as ``1``. The downloaded model will be placed here: .. code:: bash mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope_ketchup/1 && \ cp ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/Ketchup.pth ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope_ketchup/ 3. Now select a backend. The PyTorch and ONNX options **MUST** be run on ``x86_64``: - To run ONNX models with Triton, export the model into an ONNX file using the ``dope_converter`` script provided: .. code:: bash ros2 run isaac_ros_dope dope_converter.py --format onnx \ --input ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope_ketchup/Ketchup.pth --output ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope_ketchup/1/model.onnx \ --input_name INPUT__0 --output_name OUTPUT__0 --row 720 --col 1280 - To run ``TensorRT Plan`` files with Triton, first copy the generated ``onnx`` model from the above point to the target platform (e.g. a Jetson or an ``x86_64`` machine). The model will be assumed to be copied to ``${ISAAC_ROS_WS}/isaac_ros_assets/models/dope_ketchup/1/model.onnx`` inside the Docker container. Then use ``trtexec`` to convert the ``onnx`` model to a ``plan`` model: .. code:: bash /usr/src/tensorrt/bin/trtexec --onnx=${ISAAC_ROS_WS}/isaac_ros_assets/models/dope_ketchup/1/model.onnx --saveEngine=${ISAAC_ROS_WS}/isaac_ros_assets/models/dope_ketchup/1/model.plan - To run PyTorch model with Triton (**inferencing PyTorch model is supported for x86_64 platform only**), the model needs to be saved using ``torch.jit.save()``. The downloaded DOPE model is saved with ``torch.save()``. Export the DOPE model using the ``dope_converter`` script: .. code:: bash ros2 run isaac_ros_dope dope_converter.py --format pytorch \ --input ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope_ketchup/Ketchup.pth --output ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope_ketchup/1/model.pt --row 720 --col 1280 4. Create a configuration file for this model at path ``${ISAAC_ROS_WS}/isaac_ros_assets/models/dope_ketchup/config.pbtxt``. Note that name has to be the same as the model repository. Depending on the platform selected from a previous step, a slightly different ``config.pbtxt`` file must be created: ``onnxruntime_onnx`` (``.onnx`` file), ``tensorrt_plan`` (``.plan`` file) or ``pytorch_libtorch`` (``.pt`` file): .. code:: bash name: "dope_ketchup" platform: max_batch_size: 0 input [ { name: "INPUT__0" data_type: TYPE_FP32 dims: [ 1, 3, 720, 1280 ] } ] output [ { name: "OUTPUT__0" data_type: TYPE_FP32 dims: [ 1, 25, 90, 160 ] } ] version_policy: { specific { versions: [ 1 ] } } The ```` part should be replaced with ``onnxruntime_onnx`` for ``.onnx`` files, ``tensorrt_plan`` for ``.plan`` files and ``pytorch_libtorch`` for ``.pt`` files. .. note:: The DOPE decoder currently works with the output of a DOPE network that has a fixed input size of 640 x 480, which are the default dimensions set in the script. In order to use input images of other sizes, make sure to crop or resize using ROS 2 nodes from :ir_repo:`Isaac ROS Image Pipeline ` or similar packages. If another image resolution is desired, please see :doc:`here `. .. note:: The model name must be ``model.``. 5. Start ``isaac_ros_dope`` using the launch file: .. code:: bash ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py model_name:=dope_ketchup model_repository_paths:=[${ISAAC_ROS_WS}/isaac_ros_assets/models] input_binding_names:=['INPUT__0'] output_binding_names:=['OUTPUT__0'] object_name:=Ketchup .. note:: ``object_name`` should correspond to one of the objects listed in the DOPE configuration file, and the specified model should be a DOPE model that is trained for that specific object. 6. Open **another** terminal, and enter the Docker container again: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh Then, play the rosbag: .. code:: bash ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_dope/quickstart.bag 7. Open another terminal window and attach to the same container. You should be able to get the poses of the objects in the images through ``ros2 topic echo``: In a **third** terminal, enter the Docker container again: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh .. code:: bash ros2 topic echo /detections .. note:: We are echoing ``/detections`` because we remapped the original topic ``/dope/detections`` to ``detections`` in the launch file. Now visualize the detections array in RViz2: .. code:: bash rviz2 Make sure to update the ``Fixed Frame`` to ``tf_camera``. Then click on the ``Add`` button, select ``By display type`` and choose ``Detection3DArray`` under ``vision_msgs_rviz_plugins``. Expand the ``Detection3DArray`` display and change the topic to ``/detections``. Check the ``Only Edge`` option. Then click on the ``Add`` button again and select ``By Topic``. Under ``/dope_encoder``, expand the ``/resize`` drop-down, select ``/image``, and click the ``Camera`` option to see the image with the bounding box over detected objects. Refer to the pictures below. .. figure:: :ir_lfs:`` :width: 600px :align: center .. figure:: :ir_lfs:`` :width: 600px :align: center .. note:: For best results, crop/resize input images to the same dimensions your DNN model is expecting.