Isaac Sim Examples ================== .. figure:: :ir_lfs:`` :align: center :width: 600px This page contains tutorials for running :ir_repo:`Isaac ROS Nvblox ` on simulated data streaming out of Isaac Sim. The tutorial builds a reconstruction from simulated (depth) image data. The reconstruction is converted to a 2D costmap that is passed to `Nav2 `__ and used for navigation. The tutorial describes options for using :ir_repo:`Isaac ROS Visual SLAM ` and :ir_repo:`Isaac ROS DNN Stereo Depth ` for pose and depth estimation respectively, and for `reconstruction in the presence of people <#reconstruction-with-people>`_ and `other dynamic objects <#reconstruction-with-dynamic-scene-elements>`_. Hardware Requirements --------------------- Simulating the scene and robot sensors requires an RTX-enabled GPU of sufficient capability and memory capacity. In particular, we recommend an "ideal" machine in the `Isaac Sim requirements `__. .. note:: The sample scene ``localhost/NVIDIA/Assets/Isaac/4.2/Isaac/Samples/NvBlox/nvblox_sample_scene.usd`` simulates the output for a 3d lidar and three stereo cameras (depth + color images) by default. If your system is experiencing compute or memory issue while running the scene, disabling `Isaac Create Render Product `__ for some of the sensors can reduce the system load. Prerequisites ------------- These are the steps common to running all Nvblox examples in Isaac Sim. You must: 1. Complete the :doc:`/getting_started/dev_env_setup`. 2. Complete the :doc:`/getting_started/isaac_sim/index`. Install ------- 1. Complete the :ref:`nvblox quickstart `. Isaac Sim Example ----------------- This example runs a nvblox-based reconstruction with Isaac Sim supplying images, ground-truth depth, and poses from Isaac Sim. The reconstruction is used for navigation with `Nav2 `__. - **Terminal #1:** 1. Opening a terminal from the Isaac Sim launcher GUI, as described in :doc:`/getting_started/isaac_sim/index`. 2. Start the simulation by running: .. code:: bash ./isaac-sim.sh 3. Open the scene at the path ``localhost/NVIDIA/Assets/Isaac/4.2/Isaac/Samples/NvBlox/nvblox_sample_scene.usd``. 4. Play the scene to start the ROS communication from sim. - **Terminal #2:** 1. Start the Isaac ROS Dev Docker container (if not started in the install step): .. code:: bash cd $ISAAC_ROS_WS && ./src/isaac_ros_common/scripts/run_dev.sh 2. Navigate (inside the Docker) to the workspace folder and source the workspace: .. code:: bash cd /workspaces/isaac_ros-dev source install/setup.bash 3. Set this flag to ensure DDS communication with Isaac Sim runs over UDP. The file mentioned should be in the Isaac ROS docker image. .. code:: bash export FASTRTPS_DEFAULT_PROFILES_FILE=/usr/local/share/middleware_profiles/rtps_udp_profile.xml 4. Launch the example: .. code:: bash ros2 launch nvblox_examples_bringup isaac_sim_example.launch.py - **In RViz** 1. Click on the **2D Goal Pose** button. Validate that you see the mesh, costmap, and the robot moving towards the goal location, as shown at the top of this page. Selecting the Sensors --------------------- Nvblox can integrate data from 3d lidar and up to 3 cameras simultaneously. To enable the 3d lidar or more cameras you may use the ``lidar`` and ``num_cameras`` arguments by modifying the following command from `Isaac Sim Example <#isaac-sim-example>`_: - **Terminal #2:** 3. Launch the example: .. code:: bash ros2 launch nvblox_examples_bringup isaac_sim_example.launch.py \ lidar:=<"lidar"> num_cameras:=<"num_cameras"> where `"lidar"` can be set to ``True`` to enable 3d lidar and `"num_cameras"` is either 0, 1 or 3. Setting `"num_cameras"` to 1 will enable the front stereo camera (default) and setting it to 3 will enable the side stereo cameras additionally. Reconstruction With People -------------------------- .. figure:: :ir_lfs:`` :width: 600px :align: center This tutorial demonstrates how to perform dynamic people reconstruction in Nvblox using ground-truth people segmentation from Isaac Sim. For more information on how people reconstruction works, see :doc:`/concepts/scene_reconstruction/nvblox/technical_details`. To run this example modify the following commands from `Isaac Sim Example <#isaac-sim-example>`_: - **Terminal #1:** 1. Opening a terminal from the Isaac Sim launcher GUI, as described in :doc:`/getting_started/isaac_sim/index`. 2. Start the simulation by running: .. code:: bash ./isaac-sim.sh 3. Open the scene at the path ``localhost/NVIDIA/Assets/Isaac/4.2/Isaac/Samples/NvBlox/nvblox_sample_scene.usd``. 4. Toggle `/World/Humans` to choose to have the scene with people. 5. Play the scene to start the ROS communication from sim. .. note:: Because the animation requires execution of Python scripts, running the scene with the UI asks you to confirm that you want to enable script execution. Click Yes to make it possible to start the scene and the people animation. - **Terminal #2:** 3. Launch the example: .. code:: bash ros2 launch nvblox_examples_bringup isaac_sim_example.launch.py \ mode:=people_segmentation num_cameras:=1 .. note:: Multi-camera and lidar data integration is not supported in ``people_segmentation`` mode. Reconstruction With Dynamic Scene Elements ------------------------------------------ .. figure:: :ir_lfs:`` :width: 600px :align: center This tutorial demonstrates how to build a reconstruction with dynamic elements in the scene (people and non-people) using Isaac Sim data. For more information about how dynamic reconstruction works in Nvblox see :doc:`/concepts/scene_reconstruction/nvblox/technical_details`. To run this example modify the following commands from `Isaac Sim Example <#isaac-sim-example>`_: - **Terminal #1:** 1. Opening a terminal from the Isaac Sim launcher GUI, as described in :doc:`/getting_started/isaac_sim/index`. 2. Start the simulation by running: .. code:: bash ./isaac-sim.sh 3. Open the scene at the path ``localhost/NVIDIA/Assets/Isaac/4.2/Isaac/Samples/NvBlox/nvblox_sample_scene.usd``. 4. Toggle `/World/Dynamics` to choose to have the scene with dynamics. 5. Play the scene to start the ROS communication from sim. - **Terminal #2:** 3. Launch the example: .. code:: bash ros2 launch nvblox_examples_bringup isaac_sim_example.launch.py \ mode:=dynamic .. note:: Lidar data integration is not supported in dynamic mode. Running on a Custom Scene ^^^^^^^^^^^^^^^^^^^^^^^^^ To test the reconstruction on another scene: - Make sure you use the same robot USD so that the topic names and Isaac Sim ROS bridge is correctly set up. - Make sure that humans you add to the scene have the ``person`` semantic segmentation class. To do so, you can use the Semantics Schema Editor on the top prim of the additional humans. Troubleshooting --------------- See :doc:`/repositories_and_packages/isaac_ros_nvblox/isaac_ros_nvblox/troubleshooting/troubleshooting_nvblox_isaac_sim`.