============== |package_name| ============== :ir_github:` ` Quickstart ---------- Set Up Development Environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. include:: /_snippets/set_up_dev_env.rst Download Quickstart Assets ^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. Download quickstart data from NGC: :ir_assets:` ` Build |package_name| ^^^^^^^^^^^^^^^^^^^^ .. tabs:: .. tab:: Binary Package 1. Launch the Docker container using the ``run_dev.sh`` script: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 2. Install the prebuilt Debian package: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-ess && \ sudo apt-get install -y ros-humble-isaac-ros-ess-models-install 3. Download and install the pre-trained ESS model files: :ir_apt: .. code:: bash ros2 run isaac_ros_ess_models_install install_ess_models.sh --eula .. tab:: Build from Source 1. Clone this repository under ``${ISAAC_ROS_WS}/src``: .. code:: bash cd ${ISAAC_ROS_WS}/src && \ git clone :ir_clone:`` 2. Launch the Docker container using the ``run_dev.sh`` script: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 3. Use ``rosdep`` to install the package's dependencies: :ir_apt: .. code:: bash rosdep update && rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_dnn_stereo_depth/isaac_ros_ess --ignore-src -y 4. Download and install the pre-trained ESS model files: .. code:: bash cd ${ISAAC_ROS_WS} && \ colcon build --packages-up-to isaac_ros_common && \ source install/setup.bash && \ ./src/isaac_ros_dnn_stereo_depth/isaac_ros_ess_models_install/asset_scripts/install_ess_models.sh --eula 5. Build the package from source: .. code:: bash cd ${ISAAC_ROS_WS} && \ colcon build --packages-up-to isaac_ros_ess --base-paths ${ISAAC_ROS_WS}/src/isaac_ros_dnn_stereo_depth/isaac_ros_ess 6. Source the ROS workspace: .. note:: Make sure to repeat this step in **every** terminal created inside the Docker container. Since this package was built from source, the enclosing workspace must be sourced for ROS to be able to find the package's contents. .. code:: bash source install/setup.bash .. note:: Limitations on x86_64: ESS plugins only run with GPU with `sm80` and above. This limits the GPU devices on x86_64 to devices with `compute_80` and above. For CUDA compute capability details, please refer to `cuda-gpus `_. Run Launch File ^^^^^^^^^^^^^^^ .. tabs:: .. tab:: Rosbag 1. Continuing inside the Docker container, install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples 2. Run the following launch file to spin up a demo using quickstart rosbag: To run ESS at a threshold of 0.0 (fully dense output): .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=ess_disparity \ engine_file_path:=${ISAAC_ROS_WS:?}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.1.0_onnx/ess.engine \ threshold:=0.0 3. Open a **second terminal** and attach to the container: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 4. In the **second terminal**, play the ESS sample rosbag downloaded in the quickstart assets: .. code:: bash ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_ess/rosbags/ess_rosbag \ --remap /left/camera_info:=/left/camera_info_rect /right/camera_info:=/right/camera_info_rect .. tab:: RealSense Camera .. note:: This tutorial requires a compatible RealSense camera from the list of available :doc:`cameras ` #. Ensure that you have already set up your RealSense camera using the :doc:`RealSense setup tutorial `. If you have not, please set up the sensor and then restart this quickstart from the beginning. 2. Continuing inside the Docker container, install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-realsense 3. Complete steps to set up the ESS model as described in the :ref:`quickstart `. 4. Continuing inside the Docker container, run the following launch file to spin up a demo using RealSense stereo camera: To run at a threshold of 0.4: .. tabs:: .. tab:: ESS .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_stereo_rect,ess_disparity \ engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.1.0_onnx/ess.engine \ threshold:=0.4 realsense_config_file:=$(ros2 pkg prefix isaac_ros_ess --share)/config/realsense.yaml .. tab:: Light ESS .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_stereo_rect,ess_disparity \ engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.1.0_onnx/light_ess.engine \ threshold:=0.4 realsense_config_file:=$(ros2 pkg prefix isaac_ros_ess --share)/config/realsense.yaml .. tab:: Hawk Camera .. note:: This tutorial requires an Argus-compatible stereo camera from the list of available :doc:`cameras `. If you have an :ir_repo:`Argus-compatible camera `, you can also use the launch file provided in this tutorial to start a fully NITROS-accelerated stereo depth graph. #. Ensure that you have already set up your Hawk camera using the :doc:`Hawk setup tutorial `. If you have not, please set up the sensor and then restart this quickstart from the beginning. 2. Continuing inside the Docker container, install the following dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-argus-camera 3. Complete steps to set up the ESS model as described in the :ref:`quickstart `. 4. Continuing inside the Docker container, run the following launch file to spin up a demo using Argus stereo camera: To run at a threshold of 0.4: .. tabs:: .. tab:: ESS .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=argus_stereo,rectify_stereo,ess_disparity \ engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.1.0_onnx/ess.engine \ threshold:=0.4 .. tab:: Light ESS .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=argus_stereo,rectify_stereo,ess_disparity \ engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.1.0_onnx/light_ess.engine \ threshold:=0.4 .. tab:: ZED Camera #. Ensure that you have already set up your ZED camera using :doc:`ZED setup tutorial `. #. Continuing inside the Docker container, install dependencies: :ir_apt: .. code:: bash sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-depth-image-proc ros-humble-isaac-ros-stereo-image-proc ros-humble-isaac-ros-zed To run at a threshold of 0.4: .. tabs:: .. tab:: ESS .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=zed_stereo_rect,ess_disparity \ engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.1.0_onnx/ess.engine \ threshold:=0.4 zed_config_file:=$(ros2 pkg prefix isaac_ros_ess --share)/config/zed.yaml \ interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_ess/zed2_quickstart_interface_specs.json .. tab:: Light ESS .. code:: bash ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=zed_stereo_rect,ess_disparity \ engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.1.0_onnx/light_ess.engine \ threshold:=0.4 zed_config_file:=$(ros2 pkg prefix isaac_ros_ess --share)/config/zed.yaml \ interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_ess/zed2_quickstart_interface_specs.json .. note:: If you are using the `ZED X` series, replace `zed2_quickstart_interface_specs.json` with `zedx_quickstart_interface_specs.json` in the above command. Visualize Output ^^^^^^^^^^^^^^^^^^^ .. tabs:: .. tab:: Visualizer script 1. Open a **terminal** and attach to the container: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh 2. In the **terminal**, visualize and validate the disparity output using the visualizer script: .. code:: bash ros2 run isaac_ros_ess isaac_ros_ess_visualizer.py With threshold set to 0.4, the example result is: .. figure:: :ir_lfs:`` :align: center With threshold set to 0.0, the example result is: .. figure:: :ir_lfs:`` :align: center .. tab:: Foxglove Connect Foxglove Studio and setup an `Image` panel to visualize depth image using topic `/depth`. Try More Examples ----------------- To continue your exploration, check out the following suggested examples: .. toctree:: :maxdepth: 1 Tutorial with Isaac Sim Tutorial with HAWK with Wide FoV Visualize ESS Disparity Output Troubleshooting --------------- .. include:: /repositories_and_packages/isaac_ros_dnn_stereo_depth/_snippets/troubleshooting.rst Isaac ROS Troubleshooting ^^^^^^^^^^^^^^^^^^^^^^^^^ For solutions to problems with Isaac ROS, please check :doc:`here `. API --- Overview ^^^^^^^^ The ``isaac_ros_ess`` package offers functionality to generate a stereo disparity map from stereo images using a trained ESS model. Given a pair of stereo input images, the package generates a continuous disparity image for the left input image. Usage ^^^^^ .. code:: bash ros2 launch isaac_ros_ess isaac_ros_ess.launch.py engine_file_path:= ESSDisparityNode ^^^^^^^^^^^^^^^^ ROS Parameters ~~~~~~~~~~~~~~ ==================== ========== ============== =========================================================================================================================================================================================================== ROS Parameter Type Default Description ==================== ========== ============== =========================================================================================================================================================================================================== ``engine_file_path`` ``string`` N/A - Required The absolute path to the ESS engine file. ``threshold`` ``double`` 0.40 Threshold value ranges between 0.0 and 1.0 for filtering disparity with confidence. Pixels with confidence less than threshold will be marked as invalid in the disparity output. Value 0.0 means a fully dense disparity output. ``image_type`` ``string`` ``"RGB_U8"``. The input image encoding type. Supports ``"RGB_U8"`` and ``"BGR_U8"``. Note that if this parameter is not specified and there is an upstream Isaac ROS NITROS node, the type will be decided automatically. ==================== ========== ============== =========================================================================================================================================================================================================== ROS Topics Subscribed ~~~~~~~~~~~~~~~~~~~~~ ===================== ================================================================================================================= ================================= ROS Topic Interface Description ===================== ================================================================================================================= ================================= ``left/image_rect`` `sensor_msgs/Image `__ The left image of a stereo pair. ``right/image_rect`` `sensor_msgs/Image `__ The right image of a stereo pair. ``left/camera_info`` `sensor_msgs/CameraInfo `__ The left camera model. ``right/camera_info`` `sensor_msgs/CameraInfo `__ The right camera model. ===================== ================================================================================================================= ================================= .. note:: The images on input topics (``left/image_rect`` and ``right/image_rect``) should be a color image either in ``rgb8`` or ``bgr8`` format. ROS Topics Published ~~~~~~~~~~~~~~~~~~~~ ============= ========================================================================================================================= =========================================== ROS Topic Interface Description ============= ========================================================================================================================= =========================================== ``disparity`` `stereo_msgs/DisparityImage `__ The continuous stereo disparity estimation. ============= ========================================================================================================================= =========================================== Input Restrictions ~~~~~~~~~~~~~~~~~~ 1. The input left and right images must have the **same dimension and resolution**, and the resolution must be **no larger than 1920x1200**. 2. Each input pair (``left/image_rect``, ``right/image_rect``, ``left/camera_info`` and ``right/camera_info``) should have the **same timestamp**; otherwise, the synchronizing module inside the ESS Disparity Node will drop the input with smaller timestamps. Output Interpretations ~~~~~~~~~~~~~~~~~~~~~~ 1. The ``isaac_ros_ess`` package outputs a disparity image with dimension same as the ESS model output dimension. ================== ================ ESS Model Output Dimension ================== ================ ``ess.onnx`` 960 x 576 ``light_ess.onnx`` 480 x 288 ================== ================ The input images are rescaled to the ESS model input dimension before inferencing. There are two outputs from the ESS model with the same dimension: disparity output and confidence output. The disparity is filtered with confidence using a pre-configured threshold. Pixels with confidence less than the threshold is replaced with -1.0 as invalid before the inference result is published. For fully dense disparity output without confidence thresholding, set the threshold to 0.0. 2. The left and right ``CameraInfo`` are used to composite a ``stereo_msgs/DisparityImage``. If you only care about the disparity image, and don't need the baseline and focal length information, you can pass dummy camera messages. .. |package_name| replace:: ``isaac_ros_ess``