RealSense Issues

This page contains troubleshooting for the RealSense Camera Examples. The example combines nvblox-based reconstruction with VSLAM. Both of these processes, nvblox and VSLAM, are driven by the RealSense depth camera.

The setup for the example is shown below.

Images (2 x IR images and 1 x Depth image) are published at 60Hz from the RealSense ROS 2 node. At the time of publishing we are using the ros2-development branch of realsense-ros. This is the branch used by the ROS 2 apt packages (viewable on the ros package index).

The RealSense splitter: The RealSense D400 series uses an [inbuilt optical projector](https://dev.intelrealsense.com/docs/projectors) to achieve active stereo vision for improved depth quality. However, the emitted pattern visible on the stereo IR images disrupts VSLAM performance. To achieve both, improved depth quality and reliable VSLAM tracking, we configure the emitter to toggle between interleaved frames.

The RealSense splitter node is part of Isaac ROS Nvblox and filters the incoming RealSense image streams depending on the emitter state at image capture. Only infra images with the emitter off are forwarded to VSLAM and only depth images with emitter on to nvblox.

Symptoms

No reconstruction appears in RViz, and the terminal reports:

[nvblox_node-3] Warning: Invalid frame ID "odom" passed to canTransform argument target_frame - frame does not exist
[nvblox_node-3] at line 93 in /opt/ros/humble/src/geometry2/tf2/src/buffer_core.cpp
[nvblox_node-3] Warning: Invalid frame ID "odom" passed to canTransform argument target_frame - frame does not exist
[nvblox_node-3] at line 93 in /opt/ros/humble/src/geometry2/tf2/src/buffer_core.cpp

Explanation: This message is repeating because nvblox can’t get the camera pose in the odom frame. Typically, this means that VSLAM is not publishing anything because it is not receiving input data.

Typical causes are:

1. Images not coming off the camera into Linux.

2. Images not being published by the ROS node.

3. Images not making it through the splitter (usually because the metadata does not contain the projector state).

There are three different solutions.

Issue

It appears that the underlying issue is that RealSense does not support recent versions of the Linux kernel (see here for the list of supported kernel versions). There are some related issues here and here.

To determine next steps, try the following checks:

Check #1 - Images and Metadata are coming into Linux

Check that images are making it off the camera. Start the realsense-viewer.

realsense-viewer

Then:

1. Before turning on the “Stereo Module” click the drop down arrow (red box below) and turn on Infrared camera.

2. Turn on the stereo module.

3. Click the further drop down label controls and click on the option “Emitter On Off”. (purple box below)

4. In the upper right, change from 3D view to 2D. (green box below)

5. Verify that you see an image similar to the one below.

6. On the grayscale image, click the left-most icon in the upper-right-hand corner, which looks like three horizontal bars with two tiny check marks. This displays the metadata. (orange box below)

You might see two images like:

If you see the metadata labeled “Frame Laser Pose Mode” (green box above) flicking between 0 and 1 with the emitter, this is working properly. If not, preceded to the section below for Solution 1 - Install DKMS.

Check #2 - Check Images are Being Published in ROS

If your realsense-viewer is operating as expected by showing IR images and emitter-state metadata, but the nvblox, VSLAM, RealSense example is still not generating a reconstruction:

1. Start the example:

ros2 launch nvblox_examples_bringup realsense_example.launch.py

2. In RViz, verify that the following images can be viewed:

• Topics before splitter:

  • /camera/color/image_raw

  • /camera/depth/image_rect_raw

  • /camera/infra1/image_rect_raw

  • /camera/infra2/image_rect_raw

• Topics after splitter:

  • /camera/realsense_splitter_node/output/depth

  • /camera/realsense_splitter_node/output/infra_1

  • /camera/realsense_splitter_node/output/infra_2

The topics can be viewed by changing the topic field of the image topic (red box below).

3. If the topics before the splitter are not all viewable, preceded to Solution 2 - Build without CUDA - build realsense-ros without CUDA.

If only the topics after the splitter are not available and Check #1 passed, contact NVIDIA.

Solutions

Solutions to try:

• Install the RealSense DKMS

• Build without CUDA

• Both of the above

Try the solution the above checks directed you to. Repeat the checks 1 and 2.

Solution 1 - Install DKMS

If you are still getting no images off the camera, or you don’t get the correct metadata field:

1. On a supported kernel version, install librealsense-dkms. The apt package patches the kernel, such that the RealSense operates as expected.

If everything is as expected on a supported kernel and you run:

USERNAME@HOSTNAME:~/workspaces/isaac_ros-dev/src/isaac_ros_common (dev)$ uname -r
5.8.0-43-generic

USERNAME@HOSTNAME:~/workspaces/isaac_ros-dev/src/isaac_ros_common (dev)$ dkms status
librealsense2-dkms, 1.3.18, 5.8.0-43-generic, x86_64: installed
librealsense2-dkms, 1.3.18, 5.8.0-67-generic, x86_64: installed
nvidia, 510.85.02, 5.13.0-40-generic, x86_64: installed
nvidia, 510.85.02, 5.8.0-43-generic, x86_64: installed

USERNAME@HOSTNAME:/lib/udev/rules.d$ modinfo uvcvideo | grep "version:"
version: 1.1.2.realsense-1.3.18
srcversion: 964E8D3335D17053B8EEDD2

Verify that the word realsense appears. Typically, it appears after the second command, next to your version of the kernel or after the third command.

If you have a recent kernel and the words RealSense do not appear, or appear in the second command next to an old kernel version. For example:

USERNAME@HOSTNAME:~$ uname -r
5.15.0-48-generic
USERNAME@HOSTNAME:~$ dkms status
nvidia, 510.85.02, 5.15.0-48-generic, x86_64: installed
USERNAME@HOSTNAME:~$ modinfo uvcvideo | grep "version:"
version:        1.1.1
srcversion:     18D809600E7D1E107042647

The solution is to install the DKMS for recent versions of the kernel, from:

https://github.com/mengyui/librealsense2-dkms/releases/tag/initial-support-for-kernel-5.15

1. Download the .deb file named librealsense2-dkms-dkms_1.3.14_amd64.deb.

2. In the directory where you downloaded the file, run:

   sudo apt install ./librealsense2-dkms-dkms_1.3.14_amd64.deb

Verify that the following is returned:

USERNAME@HOSTNAME:~$ uname -r
5.15.0-48-generic

USERNAME@HOSTNAME:~$ dkms status
librealsense2-dkms, 1.3.14, 5.15.0-48-generic, x86_64: installed
librealsense2-dkms, 1.3.14, 5.15.0-50-generic, x86_64: installed
nvidia, 510.85.02, 5.15.0-48-generic, x86_64: installed
nvidia, 510.85.02, 5.15.0-50-generic, x86_64: installed
nvidia, 510.85.02, 5.4.0-126-generic, x86_64: installed
nvidia, 510.85.02, 5.4.0-128-generic, x86_64: installed

USERNAME@HOSTNAME:~$ modinfo uvcvideo | grep "version:"
version:        1.1.2.realsense-1.3.14
srcversion:     26234508927E0F6886C9A48

3. Repeat check 1. You should have images and metadata.

Solution 2 - Build without CUDA

If check 1 passed, you have images on your machine and metadata, but you have no images in ROS. The solution is to build librealsense without CUDA.

1. Manually build librealsense within the container without CUDA.

2. Run the following commands within the isaac_ros_common Docker container:

/opt/realsense/build-librealsense.sh -n

3. Rebuild the ROS 2 workspace:

rm -rf build/ install/ log/
colcon build --symlink-install --continue-on-error
source install/setup.bash

3. Go back and repeat check 2.

4. Verify that you have images in ROS before and after the splitter.

Still Not Working

If none of the previous solutions solved the problem, try build with:

-DFORCE_RSUSB_BACKEND=true

as reported in issue 10439.

Otherwise, file an issue, and describe the steps you have taken.