Tutorial for Multi-camera Visual SLAM Using HAWK Cameras ======================================================== .. figure:: :ir_lfs:`` :align: center :width: 600px Overview -------- This tutorial walks you through setting up Multi-camera SLAM based on :ir_repo:`Isaac ROS Visual SLAM ` with a Multiple :ir_repo:`Hawk cameras ` mounted on :doc:`Nova Carter ` or :doc:`Nova Orin Developer Kit ` Tutorial Walkthrough - Multi-camera Visual SLAM Execution ------------------------------------------------------------- .. tabs:: .. tab:: Live on Nova Carter 1. Complete the :doc:`Nova Carter setup tutorial `. 2. Complete the :doc:`Robot teleoperation tutorial ` with the required packages installed from the apt repository or built from source. 3. [Terminal 1] Clone the Isaac ROS Examples repository: .. code:: bash cd ${ISAAC_ROS_WS}/src git clone :ir_clone:`` 4. [Terminal 1] Open a new terminal and run the Docker container: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common ./scripts/run_dev.sh 5. [Terminal 1] Inside the running container, build and source the workspace: .. code:: bash cd ${ISAAC_ROS_WS} colcon build --symlink-install \ --packages-up-to nova_carter_bringup isaac_ros_multicamera_vo \ --packages-skip isaac_ros_ess_models_install isaac_ros_peoplesemseg_models_install source install/setup.bash 6. [Terminal 1] Run the launch file, which launches the multi-camera visual slam and wait for 10 seconds: .. code:: bash ros2 launch isaac_ros_multicamera_vo isaac_ros_visual_slam_multihawk_carter.launch.py use_rosbag:=False 7. [Terminal 2] Attach another terminal to the running container to execute robot teleoperation: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common ./scripts/run_dev.sh 8. [Terminal 2] Run the robot teleoperation with all sensors disabled: .. code:: bash source ${ISAAC_ROS_WS}/install/setup.bash ros2 launch nova_carter_bringup teleop.launch.py \ enable_wheel_odometry:=False \ enabled_stereo_cameras:=none \ enabled_fisheye_cameras:=none \ enabled_2d_lidars:=none \ enable_3d_lidar:=False \ run_foxglove:=False .. tab:: Live on Nova Orin Developer Kit 1. Complete the :doc:`Nova Orin Developer Kit setup tutorial ` and :doc:`Nova Orin Developer Kit sensor setup tutorial ` with the required packages installed from the apt repository or built from source. 2. [Terminal 1] Clone the Isaac ROS Examples repository: .. code:: bash cd ${ISAAC_ROS_WS}/src git clone :ir_clone:`` 3. [Terminal 1] Open a new terminal and run the Docker container: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common ./scripts/run_dev.sh 4. [Terminal 1] Inside the running container, build and source the workspace: .. code:: bash cd ${ISAAC_ROS_WS} colcon build --symlink-install \ --packages-up-to nova_developer_kit_bringup isaac_ros_multicamera_vo \ --packages-skip isaac_ros_ess_models_install isaac_ros_peoplesemseg_models_install source install/setup.bash 5. [Terminal 1] Run the launch file, which launches the multi-camera visual slam and wait for 10 seconds: .. code:: bash ros2 launch isaac_ros_multicamera_vo isaac_ros_visual_slam_multihawk_devkit.launch.py use_rosbag:=False .. tab:: On a ROSbag 1. Complete the :doc:`Quick Start guide for Isaac ROS Visual Slam `. 2. [Terminal 1] Clone the Isaac ROS Examples repository: .. code:: bash cd ${ISAAC_ROS_WS}/src git clone :ir_clone:`` 3. [Terminal 1] Open a new terminal and run the Docker container: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common ./scripts/run_dev.sh 4. Install the prebuilt dependencies: :ir_apt: .. code:: bash sudo apt-get install -y \ ros-humble-image-proc \ ros-humble-isaac-ros-h264-decoder \ ros-humble-nova-carter-description 5. [Terminal 1] Inside the running container, build and source the workspace: .. code:: bash cd ${ISAAC_ROS_WS} colcon build --symlink-install \ --packages-up-to isaac_ros_multicamera_vo \ --packages-skip isaac_ros_ess_models_install isaac_ros_peoplesemseg_models_install \ --packages-ignore nova_carter_bringup source install/setup.bash 6. [Terminal 1] Inside the running container launch the multi-camera visual odometry and wait for 10 seconds: .. code:: bash ros2 launch isaac_ros_multicamera_vo isaac_ros_visual_slam_multihawk_carter.launch.py use_rosbag:=True 7. [Terminal 2] Attach another terminal to the running container: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common ./scripts/run_dev.sh 8. [Terminal 2] Download example rosbags from the `r2b_2024 dataset `_. .. code:: bash mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets/rosbags/ cd ${ISAAC_ROS_WS}/isaac_ros_assets/rosbags/ wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/isaac/r2bdataset2024/versions/1/zip -O r2bdataset2024_1.zip unzip r2bdataset2024_1.zip 9. [Terminal 2] Replay the ``r2b_galileo`` or ``r2b_whitetunnel`` rosbag: .. code:: bash ros2 bag play ${ISAAC_ROS_WS}/isaac_ros_assets/rosbags/r2b_galileo .. note:: The rosbag contains H.264 compressed images, which means it is additionally required to decode each of the 8 FHD video streams. It is known that this may lead to performance problems on a Jetson-based system. To improve the quality of the visual odometry tracking on rosbags, it is recommended to either use a powerful workstation or slow down the playback speed using the CLI parameter ``--rate``. Tutorial Walkthrough - Visualizing the Outputs ---------------------------------------------- .. note:: Images and landmarks visualization may impact the performance of Visual Odometry. Please use visualization for debugging and demonstration purposes only. To enable landmark visualization, set the following parameters to ``True`` in the launch file: ``enable_slam_visualization``, ``enable_landmarks_view``, ``enable_observations_view`` Visualization with Foxglove Studio ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: These examples will stream the camera images in their full resolution to Foxglove Studio. This requires a substantial amount of bandwidth and is only done here for exemplary purposes. Most likely the image stream will be fairly choppy due to the large bandwidth. For a better visualization experience, we recommend only visualizing resized images or H.264 streams. 1. Complete the :ref:`foxglove setup ` guide. 2. In Foxglove Studio open the ``foxglove_layout_hawk.json`` layout file, which can be found in the :ir_repo:`isaac_ros_multicamera_vo repository `. 3. Validate that you can see a visualization of the Nova Carter robot, images of the front and right stereo cameras, image landmarks and odometry path. You should expect a visualization similar to the following: .. figure:: :ir_lfs:`` :width: 600px :alt: Foxglove visualization of the teleop outputs. :align: center Visualization with RViz2 ~~~~~~~~~~~~~~~~~~~~~~~~ 1. [Terminal 3] Attach another terminal to the running container to execute RViz2: .. code:: bash cd ${ISAAC_ROS_WS}/src/isaac_ros_common ./scripts/run_dev.sh 2. [Terminal 3] Open RViz2 from the new terminal attached to the running docker container: .. code:: bash source ${ISAAC_ROS_WS}/install/setup.bash rviz2 -d ${ISAAC_ROS_WS}/src/isaac_ros_examples/isaac_ros_multicamera_vo/rviz/hawk_multicamera_vo.rviz 3. Make sure you see the visualization of the Nova Carter robot's frame tree, images of the front and right stereo camera, image landmarks and odometry path. You should expect a visualization similar to the following: .. figure:: :ir_lfs:`` :align: center :width: 600px SLAM Visualization ~~~~~~~~~~~~~~~~~~ .. note:: Isaac ROS Visual can publish various SLAM-related topics. Specifically, during a Loop Closure event, it publishes sparse visual features to the ``/visual_slam/vis/loop_closure_cloud`` topic and the poses on the internal cuVSLAM map to the ``/visual_slam/vis/pose_graph_edges2`` topic. To enable these features, set the SLAM mode in the launch file by configuring the ``enable_localization_n_mapping`` parameter to ``true`` and activate the visualization by setting the ``enable_slam_visualization`` parameter to ``true``. SLAM-related topics can be clearly visualized when replaying the ``r2b_whitetunnel`` rosbag as the robot returns. The red dots indicate the correspondence between features in the current images and those stored in the internal map: .. figure:: :ir_lfs:`` :align: center :width: 600px