============== |package_name| ============== :ir_github:` ` Quickstart ---------- .. warning:: This package requires a model preparation process that differs significantly from those of other Isaac ROS packages. Running this quickstart requires a conversion step that must be performed on an ``x86_64`` system. It is not possible to complete this tutorial using only a Jetson device. To run on Jetson, first complete the tutorial on ``x86_64``. Then, manually copy the converted model from the ``x86_64`` host to the Jetson and repeat the tutorial on the Jetson. Set Up Development Environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. include:: /_snippets/set_up_dev_env.rst Download Quickstart Assets ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Download quickstart data from NGC: :ir_assets:` ` 2. Download the ``Ketchup.pth`` DOPE model from the official `DOPE GitHub `__ repository's model collection available `here `__. Move this file to ``${ISAAC_ROS_WS}/isaac_ros_assets/models/dope``. For example, if the model was downloaded to ``~/Downloads``: .. code:: bash mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/ && \ mv ~/Downloads/Ketchup.pth ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope Build |package_name| ~~~~~~~~~~~~~~~~~~~~ .. tabs:: .. tab:: Binary Package 1. Launch the Docker container using the ``run_dev.sh`` script: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 2. Install the prebuilt Debian package: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-dope .. tab:: Build from Source 1. Clone this repository under ``${ISAAC_ROS_WS}/src``: .. code:: bash cd ${ISAAC_ROS_WS}/src && \ git clone :ir_clone:`` 2. Launch the Docker container using the ``run_dev.sh`` script: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 3. Use ``rosdep`` to install the package's dependencies: :ir_apt: .. code:: bash rosdep update && rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_pose_estimation/isaac_ros_dope --ignore-src -y 4. Build the package from source: .. code:: bash cd ${ISAAC_ROS_WS} && \ colcon build --symlink-install --packages-up-to isaac_ros_dope --base-paths ${ISAAC_ROS_WS}/src/isaac_ros_pose_estimation/isaac_ros_dope 5. Source the ROS workspace: .. note:: Make sure to repeat this step in **every** terminal created inside the Docker container. Because this package was built from source, the enclosing workspace must be sourced for ROS to be able to find the package's contents. .. code:: bash source install/setup.bash Run Launch File ~~~~~~~~~~~~~~~ 1. Continuing inside the Docker container, convert the PyTorch model (``.pth``) to a general ONNX model (``.onnx``): .. warning:: This step must be performed on ``x86_64``. If you intend to run the model on a Jetson, you must first convert the model on an ``x86_64`` system, and then copy the output file to the corresponding location on the Jetson (``${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/Ketchup.onnx``) .. code:: bash ros2 run isaac_ros_dope dope_converter.py --format onnx \ --input ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/Ketchup.pth --output ${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/Ketchup.onnx --row 720 --col 1280 .. tabs:: .. group-tab:: Rosbag #. Continuing inside the Docker container, install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples #. Run the following launch file to spin up a demo of this package using the quickstart rosbag: .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=dope interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_dope/quickstart_interface_specs.json \ model_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/Ketchup.onnx engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/Ketchup.plan #. Open a **second** terminal inside the Docker container: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh #. Run the rosbag file to simulate an image stream: .. code:: bash ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_dope/quickstart.bag .. group-tab:: RealSense Camera #. Ensure that you have already set up your RealSense camera using the :doc:`RealSense setup tutorial `. If you have not, please set up the sensor and then restart this quickstart from the beginning. #. Continuing inside the Docker container, install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-realsense #. Run the following launch file to spin up a demo of this package using a RealSense camera: .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect,dope \ model_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/Ketchup.onnx engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/Ketchup.plan .. group-tab:: Hawk Camera #. Ensure that you have already set up your Hawk camera using the :doc:`Hawk setup tutorial `. If you have not, please set up the sensor and then restart this quickstart from the beginning. #. Continuing inside the Docker container, install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-argus-camera #. Run the following launch file to spin up a demo of this package using a Hawk camera: .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=argus_mono,rectify_mono,dope \ model_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/Ketchup.onnx engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dope/Ketchup.plan Visualize Results ~~~~~~~~~~~~~~~~~ 1. Open a **new** terminal inside the Docker container: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 2. Visualize the detection3d array in RViz2: .. code:: bash rviz2 .. tabs:: .. group-tab:: Rosbag Make sure to update the ``Fixed Frame`` to ``tf_camera``. .. group-tab:: RealSense Camera Make sure to update the ``Fixed Frame`` to ``camera_color_optical_frame``. .. group-tab:: Hawk Camera Make sure to update the ``Fixed Frame`` to ``left_cam``. 3. Then click on the ``Add`` button, select ``By display type`` and choose ``Detection3DArray`` under ``vision_msgs_rviz_plugins``. Expand the Detection3DArray display and change the topic to ``/detections``. Check the ``Only Edge`` option. Then click on the ``Add`` button again and select ``By Topic``. Under ``/dope_encoder``, expand the ``/resize`` drop-down, select ``/image``, and click the ``Camera`` option to see the image with the bounding box over detected objects. Refer to the picture below. .. figure:: :ir_lfs:`` :width: 600px :align: center .. figure:: :ir_lfs:`` :width: 600px :align: center Try More Examples ----------------- To continue your exploration, check out the following suggested examples: .. toctree:: :maxdepth: 1 Tutorial with Triton Tutorial with Custom Model Tutorial with Custom Size Verifying DOPE Accuracy .. note:: For best results, always crop or resize input images to the same dimensions your DOPE model is expecting. Use Different Models -------------------- Click :doc:`here ` for more information on how to use NGC models. Alternatively, consult the ``DOPE`` model repository to try other models. ============================================================================= ============================================================================ Model Name Use Case ============================================================================= ============================================================================ `DOPE `__ The DOPE model repository. This should be used if ``isaac_ros_dope`` is used ============================================================================= ============================================================================ Troubleshooting --------------- Isaac ROS Troubleshooting ~~~~~~~~~~~~~~~~~~~~~~~~~ For solutions to problems with Isaac ROS, please check :doc:`here `. Deep Learning Troubleshooting ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For solutions to problems with using DNN models, please check :doc:`here `. API ---- Usage ~~~~~~ Two launch files are provided to use this package. The first launch file launches ``isaac_ros_tensor_rt``, whereas the other one uses ``isaac_ros_triton``, along with the necessary components to perform encoding on images and decoding of the ``DOPE`` network's output. .. warning:: For your specific application, these launch files may need to be modified. Please consult the available components to see the configurable parameters. ====================================== ==================== Launch File Components Used ====================================== ==================== ``isaac_ros_dope_tensor_rt.launch.py`` :doc:`DnnImageEncoderNode `, :doc:`TensorRTNode `, :doc:`DopeDecoderNode ` ``isaac_ros_dope_triton.launch.py`` :doc:`DnnImageEncoderNode `, :doc:`TritonNode `, :doc:`DopeDecoderNode ` ====================================== ==================== .. warning:: There is also a ``config`` file that should be modified in ``isaac_ros_dope/config/dope_config.yaml``. DopeDecoderNode ~~~~~~~~~~~~~~~ ROS Parameters ^^^^^^^^^^^^^^ ======================== ========== ==================== ========================================================================================================================================================================================= ROS Parameter Type Default Description ======================== ========== ==================== ========================================================================================================================================================================================= ``configuration_file`` ``string`` ``dope_config.yaml`` The name of the configuration file to parse. Note: The node will look for that file name under ``isaac_ros_dope/config`` ``object_name`` ``string`` ``Ketchup`` The object class the DOPE network is detecting and the DOPE decoder is interpreting. This name should be listed in the configuration file along with its corresponding cuboid dimensions. ``enable_tf_publishing`` ``bool`` ``false`` Whether the DOPE Decoder Node will broadcast poses to the TF Tree. ``map_peak_threshold`` ``double`` ``0.1`` The minimum value of a peak in a DOPE belief map. ``rotation_y_axis`` ``double`` ``0.0`` Rotation along Y axis in degrees ``rotation_x_axis`` ``double`` ``0.0`` Rotation along X axis in degrees ``rotation_z_axis`` ``double`` ``0.0`` Rotation along Z axis in degrees ======================== ========== ==================== ========================================================================================================================================================================================= Configuration File ^^^^^^^^^^^^^^^^^^ The DOPE configuration file, which can be found at ``isaac_ros_dope/config/dope_config.yaml`` may need to modified. Specifically, you will need to specify an object type in the ``DopeDecoderNode`` that is listed in the ``dope_config.yaml`` file. The DOPE decoder node will pick the right parameters to transform the belief maps from the inference node to object poses. .. note:: The ``object_name`` should correspond to one of the objects listed in the DOPE configuration file, with the corresponding model used. .. note:: The parameters for rotation can be used to specify a rotation of the pose output by the network, it is useful when one would like to align detections between Foundation Pose and Dope. For example, for the soup can asset, the rotation would need to done along the y axis by 90 degrees. All rotation values here are in degrees. The rotation is performed in a ZYX sequence. ROS Topics Subscribed ^^^^^^^^^^^^^^^^^^^^^ ==================== ==================================================================================================================================================================== ==================================================================================== ROS Topic Interface Description ==================== ==================================================================================================================================================================== ==================================================================================== ``belief_map_array`` :ir_repo:`isaac_ros_tensor_list_interfaces/TensorList ` The tensor that represents the belief maps, which are outputs from the DOPE network. ``camera_info`` :ir_repo:`sensor_msgs/CameraInfo ` The input image camera_info. ==================== ==================================================================================================================================================================== ==================================================================================== ROS Topics Published ^^^^^^^^^^^^^^^^^^^^ =================== ================================================================================================================================ ========================================================================================================================================== ROS Topic Interface Description =================== ================================================================================================================================ ========================================================================================================================================== ``dope/detections`` `vision_msgs/Detection3DArray `__ An array of detections found by DOPE network and outputted by DOPE Decoder Node. Each detection specifies pose and bounding box dimensions. =================== ================================================================================================================================ ========================================================================================================================================== .. warning:: The DOPE network outputs one pose for each instance of an object. As such, the ``ObjectHypothesis[] results`` attribute for each element in the ``Detection3DArray`` has only one element that includes the object pose, but does not specify the ```score``. .. |package_name| replace:: ``isaac_ros_dope``