Tutorial: Running Isaac Perceptor in `Isaac Sim` ============================================================= This tutorial enables you to run autonomous navigation in Isaac Sim. The tutorial uses the Isaac Perceptor stack for local camera-based perception, using the simulated sensor outputs. Hardware Requirements --------------------- Simulating the scene and robot sensors requires an RTX-enabled GPU of sufficient capability and memory capacity. In particular, we recommend an "ideal" machine in the `Isaac Sim requirements `__. .. note:: This tutorial has been tested with Isaac Sim 4.2. Prerequisites ------------- To complete this tutorial you need to: 1. Set up your development environment by following the instructions in :doc:`getting started `. 2. Clone ``isaac_ros_common`` under ``${ISAAC_ROS_WS}/src``: .. code:: bash cd ${ISAAC_ROS_WS}/src && \ git clone :ir_clone:`` 3. Complete the :ref:`Isaac Perceptor Prerequisites `. 4. Complete the :doc:`/getting_started/isaac_sim/index`. Install ------- 1. Build/install the required packages: .. run_ros_app:: :repo_setup: :repo: nova_carter :package: nova_carter_bringup :asset_packages: isaac_ros_ess_models_install:isaac_ros_peoplesemseg_models_install:isaac_ros_peoplesemseg_models_install :asset_scripts: install_ess_models.sh:install_peoplesemsegnet_vanilla.sh:install_peoplesemsegnet_shuffleseg.sh 2. Please set this flag to disable SHM on Isaac Sim. Make sure that file exists in the docker container before running it. Please do this on all terminals. This makes sure that all DDS communication runs over UDP. .. code:: bash export FASTRTPS_DEFAULT_PROFILES_FILE=/usr/local/share/middleware_profiles/rtps_udp_profile.xml Instructions ------------ - **Terminal #1:** 1. Open a terminal from the Isaac Sim launcher GUI, as described in :doc:`/getting_started/isaac_sim/index`. 2. Start the simulation by running: .. code:: bash ./isaac-sim.sh 3. Open the scene at the path ``omniverse://localhost/NVIDIA/Assets/Isaac/4.2/Isaac/Samples/ROS2/Scenario/perceptor_navigation.usd``. 4. Play the scene to start the ROS communication from sim. - **Terminal #2:** Inside the Isaac ROS Development container, launch the example: .. code:: bash ros2 launch nova_carter_bringup navigation.launch.py \ mode:=simulation run_rviz:=True .. note:: If you want to run Perceptor with multiple cameras in Isaac Sim, you can refer to the :ref:`Multi-camera Perceptor In Isaac Sim ` section below. - **In RViz** 1. Click on the **2D Goal Pose** button. Verify that you see the robot moving towards the goal location, using the costmap produced by the camera-based 3D perception `Isaac Perceptor`. Visualizing the Outputs ------------------------ In RViz, it shows camera images streaming from the `Nova Carter` and `Isaac nvblox` mesh visualization of surrounding environments. Verify that you see a visualization similar to the image below. * The reconstructed colored voxels are uniformly reconstructed with a resolution of 5cm. * The computed distance map from `Isaac nvblox` outputs. The rainbow color spectrum reflects the proximity of each region to nearest obstacles. Regions closer to obstacle surfaces are marked in warmer colors (red, orange), while regions further away from obstacle surfaces are marked in cooler colors (blue, violet). To learn more about topics published by `Isaac nvblox`, you can refer to :doc:`nvblox ROS messages. ` .. figure:: :ir_lfs:`` :width: 700px :align: center Enable Global Navigation ------------------------ To enable global navigation on an occupancy grid map, first enable lidar data publishing from Isaac Sim. - **In the Isaac Sim UI:** 1. Stop the scene if it is running. 2. In the Stage panel, expand ``/Nova_Carter_ROS/ros_lidars``. 3. Find the following prims and check "Enabled" under **Property - Inputs**: - ``front_2d_lidar_render_product`` - ``back_2d_lidar_render_product`` 4. Play the scene again. - **Terminal #2:** 1. Launch the application: .. code:: bash ros2 launch nova_carter_bringup navigation.launch.py \ mode:=simulation run_rviz:=True \ enable_3d_lidar_localization:=False \ map_yaml_path:=${ISAAC_ROS_WS}/src/nova_carter/nova_carter_navigation/maps/perceptor_navigation.yaml 2. Next we'll send the robot to a pose on the map. In RViz, change the fixed frame to the map frame by setting the value of **Display - Global Options - Fixed Frame** to ``map``. If the occupancy map is not displayed, disable and re-enable "Map" in the Display panel. Zoom out to get a better view of the map if needed. 3. Set the initial pose of the robot in RViz or by publishing to the ``/initialpose`` topic. a. If in RViz, click the **2D Pose Estimate** button, then click and drag the robot's pose on the map. b. If by publishing to the ``/initialpose`` topic, in a terminal .. code:: bash ros2 topic pub -1 /initialpose geometry_msgs/msg/PoseWithCovarianceStamped '{ header: {stamp: {sec: 0, nanosec: 0}, frame_id: "map"}, pose: { pose: {position: {x: -9.7, y: 2.0, z: 0.0}, orientation: {w: 1.0}}, } }' This command sets the robot to the pose when the scene starts playing. If the robot has moved, update the position and orientation to the desired values. When the robot is ready, you should be able to see the robot's frame visualized as an axis within the global map. 4. Send the robot to a pose by clicking the **2D Goal Pose** button, then clicking and dragging the goal pose on the map. You should see the robot's path displayed on the map, and the robot should start moving. .. figure:: :ir_lfs:`` :width: 700px :align: center .. _multi_cam_perceptor_sim: Multi-camera Perceptor In Isaac Sim ----------------------------------- By default, when running Perceptor in Isaac Sim, Perceptor, as well as the simulation scene are configured to produce and consume data from a single Hawk stereo camera. Running with multiple cameras is possible, however requires a powerful machine. If you have a powerful machine (for example an `ideal `__ machine) you may follow these steps enable multi-camera Perceptor. - **In the Isaac Sim UI:** 1. Stop the scene if it is running. 2. In the Stage panel, expand ``/Nova_Carter_ROS``. 3. Find the following prims and check "Enabled" under **Property - Inputs**: - ``right_hawk/left_camera_render_product`` - ``right_hawk/right_camera_render_product`` - ``left_hawk/left_camera_render_product`` - ``left_hawk/right_camera_render_product`` 4. Play the scene again. - **Terminal #2:** Launch the example. *Note that here we're launching in local navigation mode, the command can be modified, as above, to run global navigation.* .. code:: bash ros2 launch nova_carter_bringup navigation.launch.py \ mode:=simulation run_rviz:=True \ stereo_camera_configuration:=front_left_right_configuration