Developer Environment

GXF Failure due to Tensor Stream Creation Failed

When launching an Isaac ROS node on a Jetson accessed remotely with X11 forwarding, X events emitted by CUDA for initializing a tensor stream are captured by the remote monitoring machine instead of the local Jetson.

Symptom

[component_container_mt-2] 2023-09-15 14:33:11.362 ERROR /workspaces/isaac_ros-dev/src/isaac_ros_image_pipeline/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorStream.cpp@104: tensor stream creation failed.
[component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/std/entity_warden.cpp@380: Failed to initialize component 00292 (stream)
[component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/core/runtime.cpp@616: Could not initialize entity 'IPOQAFVQYV_global' (E290): GXF_FAILURE
[component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/std/program.cpp@205: Failed to activate entity 00290 named IPOQAFVQYV_global: GXF_FAILURE
[component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/std/program.cpp@207: Deactivating...
[component_container_mt-2] 2023-09-15 14:33:11.396 ERROR gxf/core/runtime.cpp@1227: Graph activation failed with error: GXF_FAILURE
[component_container_mt-2] [ERROR] [1694759591.396604150] [left_decoder_node]: [NitrosContext] GxfGraphActivate Error: GXF_FAILURE
[component_container_mt-2] [INFO] [1694759591.396906843] [right_decoder_node]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_h264_decoder/share/isaac_ros_h264_decoder/EZXPATGEMM.yaml'
[component_container_mt-2] [ERROR] [1694759591.396922620] [left_decoder_node]: [NitrosNode] runGraphAsync Error: GXF_FAILURE
[component_container_mt-2] terminate called after throwing an instance of 'std::runtime_error'
[component_container_mt-2] what(): [NitrosNode] runGraphAsync Error: GXF_FAILURE

Solution

Disabling X11 forwarding will prevent X events from being intercepted by the remote machine, resolving the error.

Docker `buildx` plugin not installed

Isaac ROS Dev scripts rely on Docker configured with the buildx plugin which is no longer installed by default.

Symptom

nvidia@ubuntu:~/workspaces/isaac_ros-dev/src/isaac_ros_common$ ./scripts/run_dev.sh
Building aarch64.ros2_humble.ros1_noetic.user base as image: isaac_ros_dev-aarch64 using key aarch64.ros2_humble.ros1_noetic.user
Using configured docker search paths: /home/nvidia/workspaces/isaac_ros-dev/src/isaac_ros_common/scripts/../../isaac_ros_nitros_bridge/docker /home/nvidia/workspaces/isaac_ros-dev/src/isaac_ros_common/scripts/../docker
Using base image name not specified, using ''
Using docker context dir not specified, using Dockerfile directory
Resolved the following Dockerfiles for target image: aarch64.ros2_humble.ros1_noetic.user
/home/nvidia/workspaces/isaac_ros-dev/src/isaac_ros_common/scripts/../docker/Dockerfile.user
/home/nvidia/workspaces/isaac_ros-dev/src/isaac_ros_common/scripts/../../isaac_ros_nitros_bridge/docker/Dockerfile.ros1_noetic
/home/nvidia/workspaces/isaac_ros-dev/src/isaac_ros_common/scripts/../docker/Dockerfile.aarch64.ros2_humble
Building /home/nvidia/workspaces/isaac_ros-dev/src/isaac_ros_common/scripts/../docker/Dockerfile.aarch64.ros2_humble as image: aarch64-ros2_humble-image with base:
ERROR: BuildKit is enabled but the buildx component is missing or broken.
      Install the buildx component to build images with BuildKit:
      https://docs.docker.com/go/buildx/
Failed to build base image: isaac_ros_dev-aarch64, aborting.

Solution

From the official Docker install instructions (here), install the docker-buildx-plugin.

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt install docker-buildx-plugin

Repository cloned without pulling `git-lfs` files

Symptom

You fail building some isaac_ros packages, or some image files in the repository are empty.

Solution

Many Isaac ROS repositories are configured with Git LFS. For example, isaac_ros_visual_slam repository is configured with Git LFS to accommodate big rosbag files for test.

To check if some of the LFS files are not resolved, you can first list all the files that are supposed to be managed by Git LFS.

cd ${ISAAC_ROS_WS}/src/isaac_ros_common/ && \
  git lfs ls-files -s

It should give something like this.

Example of output of git lfs ls-files -s:

945a1a82ee - docker/tao/tao-converter-aarch64-tensorrt8.4.zip (36 KB)
cfbf5fbcee - resources/Isaac_sim_app_launcher.png (68 KB)
448d244a3e - resources/Isaac_sim_app_terminal.png (57 KB)
750f7de371 - resources/graph_read-write-speed.png (77 KB)
c29c6d6a3a - resources/isaac_ros_common_tools.png (30 KB)
87e38eea78 - resources/isaac_sim_initial_screen.png (265 KB)
e2292bc330 - resources/isaac_sim_nucleus_setup.png (221 KB)
ea8b297681 - resources/isaac_sim_ros_bridge.png (300 KB)

Now check the actual file size of those files.

git lfs ls-files -s | awk '{ print $3 }' | xargs ls -l

Example of output of git lfs ls-files -s | awk '{ print $3 }' | xargs ls -l:

-rw-rw-r- 1 jetson jetson 130 Mar 22 08:56 docker/tao/tao-converter-aarch64-tensorrt8.4.zip
-rw-rw-r- 1 jetson jetson 130 Mar 22 08:56 resources/graph_read-write-speed.png
-rw-rw-r- 1 jetson jetson 130 Mar 22 08:56 resources/isaac_ros_common_tools.png
-rw-rw-r- 1 jetson jetson 130 Mar 22 08:56 resources/Isaac_sim_app_launcher.png
-rw-rw-r- 1 jetson jetson 130 Mar 22 08:56 resources/Isaac_sim_app_terminal.png
-rw-rw-r- 1 jetson jetson 131 Mar 22 08:56 resources/isaac_sim_initial_screen.png
-rw-rw-r- 1 jetson jetson 131 Mar 22 08:56 resources/isaac_sim_nucleus_setup.png
-rw-rw-r- 1 jetson jetson 131 Mar 22 08:56 resources/isaac_sim_ros_bridge.png

In above example, all the file sizes are 130byte or 131byte, much smaller than what was advertised by git lfs ls-files -s. This shows that those LFS files are not really pulled.

Optionally, if you know a directory that should have a large LFS file, you can check the directory size.

du -hs ${ISAAC_ROS_WS}/src/isaac_ros_visual_slam/isaac_ros_visual_slam/test/test_cases/rosbags/
20K     /home/username/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam/isaac_ros_visual_slam/test/test_cases/rosbags/

If any of above is the case, you can manually pull the LFS files, after installing git lfs.

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
  git lfs pull

or

cd ${ISAAC_ROS_WS}/src/isaac_ros_visual_slam && \
  git lfs pull -X "" -I isaac_ros_visual_slam/test/test_cases/rosbags/

X11-forward issue

Symptom

When you run X11 window application inside the Isaac ROS container, you see an error message:

“X11 connection rejected because of wrong authentication.”

Solution

Go back to your docker host, and re SSH-login from your remote machine with ssh -X option.

ssh -X ${USER}@${IP}

Check if you see an error message like this.

/usr/bin/xauth:  /home/jetson/.Xauthority not writable, changes will be ignored
/usr/bin/xauth:  /home/jetson/.Xauthority not writable, changes ignored

If so, delete the ~/.Xauthority directory.

sudo rm -rf ~/.Xauthority/

Log out and re-login with ssh -X.

Check if you have .Xauthority file with your user as owner.

~$ ll ~/.Xauthority
-rw------- 1 jetson jetson 61 Mar 22 14:08 /home/jetson/.Xauthority

Now, you should be able to achieve X11 forwarding. Try with xeyes on your docker host.

Then, launch your container and try running the app that requires X11 forwarding.

ROS 2 Domain ID Collision

Symptom

You see ROS 2 topics that your system is not (yet) publishing
ROS 2 topics don’t go away even after you stop the node that you think is publishing

Solution

By default, all ROS 2 nodes use domain ID 0, so it is possible you are seeing topics from other machines on the network (docs.ros.org).

Solution 1:

You can declare ROS_DOMAIN_ID on every bash instance inside the Issac ROS container.

ROS_DOMAIN_ID=42

Solution 2:

You declare the ROS_DOMAIN_ID on the host once, and modify the run_dev.sh script to carry the environment variable.

vi ${ISAAC_ROS_WS}/src/isaac_ros_common/scripts/run_dev.sh

Find the portion that list all the environment variables to carry over, and add like this.

DOCKER_ARGS+=("-v /tmp/.X11-unix:/tmp/.X11-unix")
DOCKER_ARGS+=("-v $HOME/.Xauthority:/home/admin/.Xauthority:rw")
DOCKER_ARGS+=("-e DISPLAY")
DOCKER_ARGS+=("-e ROS_DOMAIN_ID")

No `/opt/ros/humble/install` directory in Docker container

Symptom

You expect the /opt/ros/humble/install directory to exist inside the Isaac ROS Dev Docker container, but this directory does not exist.

This is the result of a change made in the Isaac ROS 2.1 release in November 2023.

In Isaac ROS 2.1, all base ROS 2 Humble packages are installed using pre-built Debian packages produced by the NVIDIA ROS 2 Buildfarm. Previously, these base packages were built from source in a customized global underlay workspace rooted at /opt/ros/humble/.

Since the base packages are no longer built from source inline within the image, the ./install subfolder of /opt/ros/humble is never created.

Solution

Solution 1:

Move any user-specific ROS 2 packages out of the global underlay workspace to an appropriate local overlay workspace, as recommended by the official ROS documentation.

Then, build and source this overlay workspace normally.

Solution 2:

Carefully recreate the customized global underlay workspace by following the steps used in Isaac ROS 2.0 and older releases:

# EXAMPLE: Install OSRF negotiated ROS 2 package from source

ENV ROS_DISTRO=humble
ENV ROS_ROOT=/opt/ros/${ROS_DISTRO}

RUN apt-get update && mkdir -p ${ROS_ROOT}/src && cd ${ROS_ROOT}/src \
   && git clone https://github.com/osrf/negotiated && cd negotiated && git checkout master \
   && source ${ROS_ROOT}/setup.bash \
   && cd negotiated_interfaces && bloom-generate rosdebian && fakeroot debian/rules binary \
   && cd ../ && apt-get install -y ./*.deb && rm ./*.deb \
   && cd negotiated && bloom-generate rosdebian && fakeroot debian/rules binary \
   && cd ../ && apt-get install -y ./*.deb && rm ./*.deb \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get clean

This process involves overriding the behavior of bloom-generate to force it to produce and install the Debian packages inline.

Developer Environment

GXF Failure due to Tensor Stream Creation Failed

Symptom

Solution

Docker buildx plugin not installed

Symptom

Solution

Repository cloned without pulling git-lfs files

Symptom

Solution

X11-forward issue

Symptom

Solution

ROS 2 Domain ID Collision

Symptom

Solution

No /opt/ros/humble/install directory in Docker container

Symptom

Solution

Docker `buildx` plugin not installed

Repository cloned without pulling `git-lfs` files

No `/opt/ros/humble/install` directory in Docker container