isaac_ros_sipl_camera

Source code available on GitHub.

Quickstart

Set Up Development Environment

1. Set up your development environment by following the instructions in getting started.

2. (Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

   Note

   We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the development environment during installation, which will interrupt the quickstart process.

The SIPL headers are expected to be installed on the host with NVIDIA SDK Manager when flashing the JetPack BSP. When installing with SDK Manager, select all available optional packages. For installation instructions, see the JetPack Installation Guide.

If JetPack 7.1 was installed using an ISO image or with OTA updates, set CUDA environment variables and install the SIPL and Multimedia APIs:

wget https://developer.nvidia.com/downloads/embedded/L4T/r38_Release_v4.0/release/Jetson_SIPL_API_R38.4.0_aarch64.tbz2
wget https://developer.nvidia.com/downloads/embedded/L4T/r38_Release_v4.0/release/Jetson_Multimedia_API_R38.4.0_aarch64.tbz2
sudo tar xjf Jetson_SIPL_API_R38.4.0_aarch64.tbz2 -C /
sudo tar xjf Jetson_Multimedia_API_R38.4.0_aarch64.tbz2 -C /

Supported Cameras

As of JetPack 7.1 release, SIPL supports only the Camera-over-Ethernet (CoE) Leopard Imaging Eagle VB1940 stereo camera (2560x1984 at 30 fps).

Build isaac_ros_sipl_camera

Binary PackageBuild from Source

1. Activate the Isaac ROS environment:

   isaac-ros activate
   

2. Install the prebuilt Debian package:

   sudo apt-get update
   

   sudo apt-get install -y ros-jazzy-isaac-ros-sipl-camera
   

Run Launch File

Monocular CameraStereo CameraStereo Camera with RGB8 Conversion and Rectification

1. Launch the monocular camera node with default native NV12 output:

   ros2 launch isaac_ros_sipl_camera sipl_camera.launch.py
   

Visualize Results

1. Open a new terminal and activate the Isaac ROS environment:

   isaac-ros activate
   

2. Verify that the camera topics are publishing:

   ros2 topic list
   ros2 topic hz /left/camera_info
   

   Example output for stereo with format conversion and rectification enabled:

   /left/camera_info
   /left/camera_info/nitros
   /left/camera_info_rect
   /left/camera_info_rect/nitros
   /left/image_converted
   /left/image_converted/nitros
   /left/image_raw
   /left/image_raw/nitros
   /left/image_rect
   /left/image_rect/nitros
   /right/camera_info
   /right/camera_info/nitros
   /right/camera_info_rect
   /right/camera_info_rect/nitros
   /right/image_converted
   /right/image_converted/nitros
   /right/image_raw
   /right/image_raw/nitros
   /right/image_rect
   /right/image_rect/nitros
   /tf_static
   average rate: 28.127
   min: 0.000s max: 0.334s std dev: 0.05718s window: 30
   ...
   

3. Visualize the stereo camera streams with your preferred tool:

   rqt_image_view

   sudo apt-get install -y ros-jazzy-rqt-image-view
   ros2 run rqt_image_view rqt_image_view /left/image_raw
   

   Foxglove Studio

   sudo apt-get install -y ros-jazzy-foxglove-bridge
   ros2 run foxglove_bridge foxglove_bridge --ros-args \
     -p port:=9090 \
     -p send_buffer_limit:=10000000 \
     -p max_qos_depth:=5 \
     -p use_compression:=false
   

   This configuration is recommended for high-rate camera streams: max_qos_depth bounds per-topic queue growth, and send_buffer_limit helps avoid dropped messages under burst load.

   Open Foxglove Studio, connect to your ROS 2 bridge, and add Image panels for /left/image_raw and /right/image_raw.

   RViz2

   sudo apt-get install -y ros-jazzy-rviz2
   rviz2
   

   In RViz2, add Image displays and set the topics to /left/image_raw and /right/image_raw.

Data Flow

flowchart TD CoECamera["CoE Camera (VB1940)"] -->|"Ethernet (CoE)"| ISP["HW accelerated ISP"] ISP -->|"NvSciBuf (NVMM buffer)"| SiplCameraNode["SiplCameraNode"] SiplCameraNode -->|"NITROS (CUDA buffer)"| Converter["ImageFormatConverter"] Converter -->|"NITROS (CUDA buffer)"| Downstream["Downstream Nodes
(SLAM, Rectify, ESS, etc.)"]

Package Architecture

Nodes

SiplCameraNode

Base node for monocular cameras. See ROS Topics Published and ROS Parameters for the full interface specification.

SiplStereoCameraNode

Stereo camera node that extends SiplCameraNode. See ROS Topics Published and ROS Parameters for the full interface specification.

When running launch files, additional topics may be published depending on the launch arguments. See the launch files section below for more details.

Launch Files

The sipl_camera.launch.py and sipl_stereo_camera.launch.py launch files conditionally add format conversion and rectification nodes based on launch arguments. These nodes are from the isaac_ros_image_proc package.

When encoding_desired is set to a non-native format (anything other than nv12 or nv24), the launch file adds an ImageFormatConverterNode to convert the output.

When enable_rectify is set to true, the launch file adds a RectifyNode and automatically forces encoding_desired to rgb8 if a native format (nv12/nv24) was specified, because RectifyNode requires a standard image encoding.

sipl_camera.launch.py

Launch Argument	Default	Description
param_file	eagle_mono.yaml	Path to the YAML parameter file
encoding_desired	"" (native)	Output encoding: nv12/nv24 (native) or rgb8, mono8, etc. (converted)
target_container	sipl_camera_container	Component container name
enable_rectify	false	Enable image rectification through RectifyNode

Pipeline variants:

• Native output (default): SiplCameraNode

• Format conversion: SiplCameraNode → ImageFormatConverterNode

• Rectification: SiplCameraNode → ImageFormatConverterNode → RectifyNode

sipl_stereo_camera.launch.py

Launch Argument	Default	Description
param_file	eagle_stereo.yaml	Path to the YAML parameter file
encoding_desired	"" (native)	Output encoding: nv12/nv24 (native) or rgb8, mono8, etc. (converted)
target_container	sipl_camera_container	Component container name
enable_rectify	false	Enable image rectification through RectifyNode

Pipeline variants:

• Native output (default): SiplStereoCameraNode

• Format conversion: SiplStereoCameraNode → ImageFormatConverterNode (left and right)

• Rectification: SiplStereoCameraNode → ImageFormatConverterNode (left and right) → RectifyNode (left and right)

TF Frame Tree

The default frame names produce the following tree structures.

flowchart TD Camera["camera"] --> CameraOptical["camera_optical"] StereoCamera["stereo_camera"] --> LeftCamera["stereo_camera_left"] LeftCamera --> LeftOptical["stereo_camera_left_optical"] StereoCamera --> RightCamera["stereo_camera_right"] RightCamera --> RightOptical["stereo_camera_right_optical"]

Verifying Stereo Camera Calibration

Given the camera’s calibration data, you may use the epipolar test isaac_ros_rectify_stereo_epipolar_test.py in isaac_ros_image_proc package to validate that your stereo camera calibration produces accurately rectified images based on epipolar line tests.

Several files are required to run the test and should be placed in the rectify_stereo test cases directory of the isaac_ros_image_proc package.

${ISAAC_ROS_WS}/src/isaac_ros_image_pipeline/isaac_ros_image_proc/test/test_cases/rectify_stereo/

1. Refer to the quickstart section for the isaac_ros_image_proc package on how to clone and build the package.

2. Capture one left/right raw image pair of PNG files that contains a visible checkerboard pattern. For example, image_raw_right.png

3. Create test configuration JSON files for the left and right camera respectively: image_raw_left.json and image_raw_right.json. These files should reference the PNG files and include the checkerboard inner-corner grid size (rows x columns). For example, image_raw_right.json

   Note

   The checkerboard size must use inner-corner counts (for example, a 10x7 square grid uses a 9x6 inner-corner size).

4. Create camera info JSON files for left and right camera respectively: camera_info_left.json and camera_info_right.json. For example, camera_info_right.json

5. Place all the PNG and JSON files in the rectify_stereo test cases directory of the isaac_ros_image_proc package:

   # From the workspace root
   mkdir ${ISAAC_ROS_WS}/src/isaac_ros_image_pipeline/isaac_ros_image_proc/test/test_cases/rectify_stereo/sipl_stereo_camera
   cp <your_files> ${ISAAC_ROS_WS}/src/isaac_ros_image_pipeline/isaac_ros_image_proc/test/test_cases/rectify_stereo/sipl_stereo_camera
   

6. Build the isaac_ros_image_proc package after placing all the files:

   colcon build --packages-up-to isaac_ros_image_proc
   source install/setup.bash
   

7. Run the test:

   colcon test --packages-select isaac_ros_image_proc \
       --event-handlers console_direct+ \
       --ctest-args -R isaac_ros_rectify_stereo_epipolar_test
   

8. Verify the test results. Passing the test indicates that the calibration accuracy is satisfactory. If the test fails, downstream packages that consume the rectified images may still function, but with degraded performance depending on your application’s requirements.

Note

A well-rectified pair has horizontal epipolar lines with sub-pixel mean error (less than 0.5 px) and max error under approximately 1-2 px.

Below are example outputs of the test with the epipolar lines drawn.

Left Rectified

Right Rectified

API

Usage

ros2 launch isaac_ros_sipl_camera sipl_camera.launch.py param_file:=<path_to_yaml> encoding_desired:=<encoding>

ros2 launch isaac_ros_sipl_camera sipl_stereo_camera.launch.py param_file:=<path_to_yaml> encoding_desired:=<encoding> enable_rectify:=<true|false>

ROS Topics Published

SiplCameraNode

ROS Topic	Interface	Description
image_raw	sensor_msgs/Image	Raw camera image in nv12 or nv24 encoding.
camera_info	sensor_msgs/CameraInfo	Camera intrinsic calibration data.
/tf_static	tf2_msgs/TFMessage	Static transform from camera link to optical frame.

SiplStereoCameraNode

ROS Topic	Interface	Description
left/image_raw	sensor_msgs/Image	Left camera raw image in nv12 or nv24 encoding.
right/image_raw	sensor_msgs/Image	Right camera raw image in nv12 or nv24 encoding.
left/camera_info	sensor_msgs/CameraInfo	Left camera intrinsic calibration data.
right/camera_info	sensor_msgs/CameraInfo	Right camera intrinsic calibration data.
/tf_static	tf2_msgs/TFMessage	Static transforms from stereo rig base to left/right optical frames.

ROS Parameters

Common Parameters

ROS Parameter	Type	Default	Description
platform_config	string	VB1940_Camera	SIPL platform configuration name
hsb_name	string	hsb_0	Holoscan Sensor Bridge (HSB) name
hsb_id	int	0	Holoscan Sensor Bridge ID
interface_name	string	mgbe0_0	Network interface for the CoE camera
ip_address	string	""	Camera IP override (empty uses the platform default)
mac_address	string	""	Camera MAC override (empty uses the platform default)
encoding_desired	string	nv12	Native ISP output encoding: nv12 or nv24
use_hw_timestamp	bool	true	Use hardware TSC timestamps
camera_info_url	string	""	Path to camera calibration file (left camera in stereo mode)

Monocular Parameters

ROS Parameter	Type	Default	Description
camera_link_frame_name	string	camera	Camera link frame name

Stereo Parameters

ROS Parameter	Type	Default	Description
camera_link_frame_name	string	stereo_camera	Stereo rig base frame
left_camera_frame_name	string	stereo_camera_left	Left camera frame
right_camera_frame_name	string	stereo_camera_right	Right camera frame
right_camera_info_url	string	""	Path to right camera calibration file

Camera Calibration Overrides

Provide calibration files using package:// or file:// URLs. Example calibration files are available in the package config/ directory for quick testing.

The following example shows how to set the calibration file in a YAML parameter file:

sipl_camera:
  ros__parameters:
    camera_info_url: "package://isaac_ros_sipl_camera/config/mock_mono_camera_info.yaml"

Prerequisites

Environment

When you use isaac-ros activate, the environment for SIPL development will be prepared automatically.

If you need to build on the host directly, you may need to also install the SIPL headers /usr/src/jetson_sipl_api. For installation instructions, see the SIPL Application Development Guide.

The following paths and libraries are necessary for building and running SIPL.

Path	Contents	Needed For
/usr/lib/aarch64-linux-gnu/nvidia	Core SIPL runtime libraries (libnvsipl*.so, libnvsci*.so)	Runtime libraries for SIPL
/usr/src/jetson_sipl_api	SIPL development headers	Compilation
/usr/lib/nvsipl_drv	Camera sensor driver plugins	Runtime (camera detection)
/usr/lib/nvsipl_uddf	UDDF driver libraries	Runtime
/var/nvidia/nvcam	NITO platform configuration files (.nito files)	Runtime (camera ISP initialization)
/dev/coe-chan-*	CoE device nodes	Runtime (camera device files)

Library	Description	Location (Jetson Thor)
libnvsipl.so	SIPL camera API	/usr/lib/aarch64-linux-gnu/nvidia/
libnvsipl_query.so	SIPL query API	/usr/lib/aarch64-linux-gnu/nvidia/
libnvsipl_pipeline.so	SIPL pipeline API	/usr/lib/aarch64-linux-gnu/nvidia/
libnvscibuf.so	NvSci buffer	/usr/lib/aarch64-linux-gnu/nvidia/
libnvscisync.so	NvSci sync	/usr/lib/aarch64-linux-gnu/nvidia/
libnvscicommon.so	NvSci common	/usr/lib/aarch64-linux-gnu/nvidia/

Troubleshooting

Isaac ROS Troubleshooting

For solutions to problems with Isaac ROS, see troubleshooting.

SIPL CoE Camera Not Found

For any issues with CoE camera detection, see the Eagle CoE Stereo Camera Setup guide.

HSB fails to initialize with error “Address already in use”

Symptom

Unable to bind to initialize HSB and configure the HSB transport driver. You see the following error message:

[HSB] Error configuring driver: bind failed with errno=98: "Address already in use"

Solution

Another Holoscan instance may still be running. Check and stop it:

ps aux | grep hololink

If there is output, kill the process if it is safe to do so:

kill -9 <pid>

SIPL Headers or Libraries Not Found

Symptom

You see the following error message when building the package:

CMake Error: SIPL headers not found at: /usr/src/jetson_sipl_api/sipl/include

Solution

1. Install the SIPL libraries through NVIDIA SDK Manager.

2. Verify that the SIPL libraries and headers are present:

   ls /usr/lib/aarch64-linux-gnu/nvidia/libnvsipl*
   ls /usr/src/jetson_sipl_api/sipl/include/
   

Invalid MAC or IP Address Format

Symptom

Invalid MAC address format, expected XX:XX:XX:XX:XX:XX
Invalid IP address format, expected XXX.XXX.XXX.XXX

Solution

• Make sure to retrieve the CoE camera’s specific MAC and IP addresses. Refer to the Eagle CoE Stereo Camera Setup guide for instructions.

• Use colon-separated hex format for MAC addresses: 8c:1f:64:6d:70:03

• Use dotted decimal format for IP addresses: 192.168.1.2

• Verify that the addresses match the camera documentation or EEPROM data.

HSB ID Mismatch

Symptom

No matching transport found for override: hsb_id=X
No matching camera found for override: hsb_id=X

Solution

1. Verify that the hsb_id parameter matches the platform configuration.

2. Check the HSB ID in the camera system config.

3. For VB1940, the default HSB ID is typically 0.

4. Verify that the HSB ID matches across transport and camera configuration.