RealSense Issues
This page contains troubleshooting for the RealSense Camera Examples. The example combines nvblox-based reconstruction with VSLAM. Both of these processes, nvblox and VSLAM, are driven by the RealSense depth camera.
The setup for the example is shown below.
Images (2 x IR images and 1 x Depth image) are published at 60Hz from
the RealSense ROS 2 node. At the time of publishing we are using the
ros2-development
branch of
realsense-ros.
This is the branch used by the ROS 2 apt packages (viewable on the ros
package
index).
Why do we include the RealSense splitter?: The depth output from the RealSense is significantly improved when using its inbuilt projector which increases the amount of texture in the scene. However, VSLAM performance is disrupted by the projector. We, therefore, use the RealSense’s toggling feature such that the projector is switched on and off in interleaved frames (on, off, on, off, on, off,…etc.). The splitter splits this interleaved image stream into two channels, one with the projector on and one with the projector off. IR images with the projector off are passed to VSLAM, while depth images with the projector on are passed to nvblox. This splitting occurs based on the metadata associated with each frame, published by the RealSense, which contains a field which states if the projector was on at the time of frame capture.
Symptoms
No reconstruction appears in RViz, and the terminal reports:
[nvblox_node-3] Warning: Invalid frame ID "odom" passed to canTransform argument target_frame - frame does not exist
[nvblox_node-3] at line 93 in /opt/ros/humble/src/geometry2/tf2/src/buffer_core.cpp
[nvblox_node-3] Warning: Invalid frame ID "odom" passed to canTransform argument target_frame - frame does not exist
[nvblox_node-3] at line 93 in /opt/ros/humble/src/geometry2/tf2/src/buffer_core.cpp
Explanation: This message is repeating because nvblox can’t get the
camera pose in the odom
frame. It can’t get the pose because VSLAM is
not publishing it. VSLAM is not publishing anything because (likely) it
is not receiving input data.
So the issue here is that VSLAM is not getting data. This is caused either by one of: 1) Images not coming off the camera into Linux. 2) Images not being published by the ROS node. 3) Images not making it through the splitter (usually because the metadata does not contain the projector state).
I have seen all three of these issues. All three of them have the same symptom (no pose for nvblox), but all three have different solutions.
Issue
From what I can tell, the underlying issue is that the RealSense does not support recent versions of the Linux kernel (see here for the list of supported kernel versions). There are some related issues here and here.
To determine what to do we have to do a few checks.
Check #1 - Images and Metadata are coming into Linux
Check that images are making it off the camera. Start the
realsense-viewer
.
realsense-viewer
Then:
Before turning on the “Stereo Module” click the drop down arrow (red box below) and turn on Infrared camera.
Turn on the stereo module
Click the further drop down labels “controls” and click on the option “Emitter On Off”. (purple box below)
In the upper right change from 3D view to 2D. (green box below)
You should see something like the image below.
On the grayscale image click the left-most icon in the upper-right-hand corner which looks like three horizontal bars with two tiny check marks. This displays the meta data. (orange box below)
Do you see two images like below?
Do you see the metadata labeled “Frame Laser Pose Mode” (green box above) which should be flicking between 0 and 1 with the emitter. If no, preceded to the section below for Solution 1 - Install DKMS
Check #2 - Check Images are Being Published in ROS
You’re here because your realsense-viewer
is operating as expected,
i.e. is showing IR images and emitter-state metadata, but the nvblox,
VSLAM, RealSense example still is not generating a reconstruction.
Start the example:
ros2 launch nvblox_examples_bringup realsense_example.launch.py
In RViz check that the following images can be viewed.
Topics before splitter
/camera/color/image_raw
/camera/depth/image_rect_raw
/camera/infra1/image_rect_raw
/camera/infra2/image_rect_raw
Topics after splitter
/camera/realsense_splitter_node/output/depth
/camera/realsense_splitter_node/output/infra_1
/camera/realsense_splitter_node/output/infra_2
The topics can be viewed by changing the topic field of the image topic (red box below)
If the topics before the splitter are not all viewable, preceded to
Solution 2 - Build without CUDA - build realsense-ros
without CUDA.
If only the topics after the splitter are not available and Check #1 passed. I’m out of ideas and you should probably get in contact.
Solutions
The solutions I have found boil down to:
Install the RealSense DKMS, or
Build without CUDA, or
Both of the above.
Try the solution the above checks directed you to. Repeat the checks 1 and 2. If you still have no luck, attempt the other solution.
Solution 1 - Install DKMS
You’re here because either you get no images off the camera, or you don’t get the correct metadata field.
On a supported kernel version, the apt package patches the kernel when
you install librealsense-dkms
such that the RealSense operates as
expected.
If everything is as expected on a supported kernel and you run
USERNAME@HOSTNAME:~/workspaces/isaac_ros-dev/src/isaac_ros_common (dev)$ uname -r
5.8.0-43-generic
USERNAME@HOSTNAME:~/workspaces/isaac_ros-dev/src/isaac_ros_common (dev)$ dkms status
librealsense2-dkms, 1.3.18, 5.8.0-43-generic, x86_64: installed
librealsense2-dkms, 1.3.18, 5.8.0-67-generic, x86_64: installed
nvidia, 510.85.02, 5.13.0-40-generic, x86_64: installed
nvidia, 510.85.02, 5.8.0-43-generic, x86_64: installed
USERNAME@HOSTNAME:/lib/udev/rules.d$ modinfo uvcvideo | grep "version:"
version: 1.1.2.realsense-1.3.18
srcversion: 964E8D3335D17053B8EEDD2
The important thing to see here is that the word realsense
appears: -
after the second command, it appears next to your version of the
kernel, and - after the third command, it appears at all.
Likely, if you’re here, you have a recent kernel and the words RealSense do not appear, or appear in the second command next to an old kernel version. Something like:
USERNAME@HOSTNAME:~$ uname -r
5.15.0-48-generic
USERNAME@HOSTNAME:~$ dkms status
nvidia, 510.85.02, 5.15.0-48-generic, x86_64: installed
USERNAME@HOSTNAME:~$ modinfo uvcvideo | grep "version:"
version: 1.1.1
srcversion: 18D809600E7D1E107042647
The solution is to install the DKMS for recent versions of the kernel. We found these here:
https://github.com/mengyui/librealsense2-dkms/releases/tag/initial-support-for-kernel-5.15
Download the .deb file named
librealsense2-dkms-dkms_1.3.14_amd64.deb
In the directory where you
downloaded the file run:
sudo apt install ./librealsense2-dkms-dkms_1.3.14_amd64.deb
Now the
commands above, on my machine, return:
USERNAME@HOSTNAME:~$ uname -r
5.15.0-48-generic
USERNAME@HOSTNAME:~$ dkms status
librealsense2-dkms, 1.3.14, 5.15.0-48-generic, x86_64: installed
librealsense2-dkms, 1.3.14, 5.15.0-50-generic, x86_64: installed
nvidia, 510.85.02, 5.15.0-48-generic, x86_64: installed
nvidia, 510.85.02, 5.15.0-50-generic, x86_64: installed
nvidia, 510.85.02, 5.4.0-126-generic, x86_64: installed
nvidia, 510.85.02, 5.4.0-128-generic, x86_64: installed
USERNAME@HOSTNAME:~$ modinfo uvcvideo | grep "version:"
version: 1.1.2.realsense-1.3.14
srcversion: 26234508927E0F6886C9A48
Go back and repeat check 1. You should have images + metadata.
Solution 2 - Build without CUDA
You’re here because check 1 passed; you have images on your machine +
metadata, but you get no images in ROS. The solution here is to build
librealsense
without CUDA.
Manually build librealsense
within the container without CUDA. Run the
following commands within the isaac_ros_common
docker container:
/opt/realsense/build-librealsense.sh -n
now rebuild the ROS 2 workspace
rm -rf build/ install/ log/
colcon build --symlink-install --continue-on-error
source install/setup.bash
Go back and repeat check 2. You should have images in ROS before and after the splitter.
Things still are not working
Well this is not going well. One thing to try is to build with
-DFORCE_RSUSB_BACKEND=true
as reported in this
issue.
Otherwise, file an issue, and describe the steps you have taken here.