isaac_ros_teleop

Source code available on GitHub.

Quickstart

Prerequisites

This quickstart has been tested and qualified on Jetson AGX Thor and x86_64.

• PICO 4 Ultra or Meta Quest 3 headset

• If no XR headset is available, the emulator provided by Isaac Teleop Core can be used instead

• A stable network connection meeting the CloudXR network requirements

Set Up Development Environment

1. Set up your development environment by following the instructions in getting started.

2. (Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

   Note

   We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the development environment during installation, which will interrupt the quickstart process.

Build isaac_ros_teleop

1. Install and build isaac_ros_teleop:

   Binary PackageBuild from Source

   1. Activate the Isaac ROS environment:

      isaac-ros activate
      

   2. Install the prebuilt Debian package:

      sudo apt-get update
      

      sudo apt-get install -y ros-jazzy-isaac-ros-teleop
      

Prepare CloudXR

1. Whitelist firewall ports for CloudXR.

   CloudXR requires certain network ports to be open. Depending on your firewall configuration, you may need to open them manually. For Quest and PICO headsets (WebXR Client), at the minimum, the CloudXR runtime and WebSocket Secure proxy ports must be open. If ufw is enabled, run:

   sudo ufw allow 47998/udp
   sudo ufw allow 49100,48322/tcp
   

   If you are running the WebXR client from source and ufw is enabled, also open the web server ports:

   sudo ufw allow 8080,8443/tcp
   

Run Launch File

1. Run the following launch file to spin up a demo of this package:

   ros2 launch isaac_ros_teleop isaac_ros_teleop.launch.py \
       cloudxr_accept_eula:=true
   

2. After the ROS launch command is running, connect the XR headset to the teleop server. In your headset’s browser, open the CloudXR web client and follow the headset connection guide.

   Note

   If the CloudXR runtime is running on Jetson, make sure to set the Video Codec to H.264, otherwise the headset will fail to connect.

   Warning

   The world frame of the headset is defined as the position of the headset and controllers at the moment of connection. Stand still and face the robot before connecting to establish a consistent world frame. To reset the world frame, disconnect and reconnect the headset while stationary.

Visualize Results

1. Open a new terminal and activate the Isaac ROS environment:

   isaac-ros activate
   

2. Install RViz:

   sudo apt install -y ros-jazzy-rviz2
   source /opt/ros/jazzy/setup.bash
   

3. Open RViz:

   rviz2
   

   Then set the Fixed Frame to world. Then click on the Add button, select By display type, and choose TF under rviz_default_plugins.

4. With the controllers in your hands, start moving them. You should see the frames track your movements:

   

   Note

   The frames are converted to match ROS frame convention. Therefore, the poses are re-oriented from Right-Up-Back orientation to Forward-Left-Up for the x, y and z axes respectively.

Controller Reference

The PICO 4 Ultra headset and Meta Quest 3 include two handheld controllers. The following table summarizes what each input does during teleoperation:

Input	Action
Left joystick	Move the robot: up = forward, down = backward, left = strafe left, right = strafe right
Right joystick — left / right	Rotate the robot in place (yaw)
Controller motion (6-DOF)	The end-effector pose tracks the physical controller; moving and rotating the controller moves the robot’s hand correspondingly
Triggers (each controller has two)	Open and close the finger joints of the tri-finger hand

Troubleshooting

Isaac ROS Troubleshooting

For solutions to problems with Isaac ROS, see troubleshooting.

API

Usage

ros2 launch isaac_ros_teleop isaac_ros_teleop.launch.py \
    cloudxr_accept_eula:=true

Launch Arguments

Launch Argument	Type	Default	Description
ee_pose_topic	string	xr_teleop/ee_poses	Topic name for published end-effector poses (geometry_msgs/PoseArray).
root_twist_topic	string	xr_teleop/root_twist	Topic name for published root velocity command (geometry_msgs/TwistStamped).
root_pose_topic	string	xr_teleop/root_pose	Topic name for published root pose command (geometry_msgs/PoseStamped).
head_pose_topic	string	xr_teleop/head_pose	Topic name for published head pose command (geometry_msgs/PoseStamped).
finger_joints_topic	string	xr_teleop/finger_joints	Topic name for retargeted G1 TriHand finger joints (sensor_msgs/JointState).
controller_data_topic	string	xr_teleop/controller_data	Topic name for raw msgpack-encoded controller state (std_msgs/ByteMultiArray).
rate_hz	double	60.0	The publishing rate in Hz.
world_frame	string	world	World frame for message headers and TF parent frame.
right_wrist_frame	string	right_wrist	TF child frame name for the right wrist.
left_wrist_frame	string	left_wrist	TF child frame name for the left wrist.
head_frame	string	head	TF child frame name for the head.
left_finger_joint_names	string	["left_hand_thumb_0_joint","left_hand_thumb_1_joint","left_hand_thumb_2_joint","left_hand_index_0_joint","left_hand_index_1_joint","left_hand_middle_0_joint","left_hand_middle_1_joint"]	Left-hand finger joint names in TriHand retargeter order: thumb_rotation, thumb_proximal, thumb_distal, index_proximal, index_distal, middle_proximal, middle_distal.
right_finger_joint_names	string	["right_hand_thumb_0_joint","right_hand_thumb_1_joint","right_hand_thumb_2_joint","right_hand_index_0_joint","right_hand_index_1_joint","right_hand_middle_0_joint","right_hand_middle_1_joint"]	Right-hand finger joint names in TriHand retargeter order: thumb_rotation, thumb_proximal, thumb_distal, index_proximal, index_distal, middle_proximal, middle_distal.
transform_rotation	string	[0.5, -0.5, -0.5, 0.5]	Quaternion [qx, qy, qz, qw] to rotate XR poses into the ROS world frame. Defaults to the Right-Up-Back orientation to Forward-Left-Up.
transform_translation	string	[0.0, 0.0, 0.0]	Translation [x, y, z] applied to XR poses after rotation.
cloudxr_install_dir	string	${ISAAC_ROS_WS}/.cloudxr	CloudXR runtime directory used by the in-process launcher.
cloudxr_env_config	string	empty	Optional CloudXR environment config file passed to the in-process launcher.
cloudxr_accept_eula	bool	false	Accept the NVIDIA CloudXR EULA non-interactively.
cloudxr_setup_oob	bool	false	Enable the CloudXR out-of-band teleop control hub.
cloudxr_usb_local	bool	false	Route teleop traffic over USB through adb reverse.

ROS Topics Published

ROS Topic	Interface	Description
ee_pose_topic	geometry_msgs/PoseArray	The poses of the wrists or end-effectors, given in order of left wrist and then right wrist.
root_twist_topic	geometry_msgs/TwistStamped	The root twist command.
root_pose_topic	geometry_msgs/PoseStamped	The root pose command.
head_pose_topic	geometry_msgs/PoseStamped	The headset pose in the configured world frame.
finger_joints_topic	sensor_msgs/JointState	The retargeted finger joint angles for the robot.
controller_data_topic	std_msgs/ByteMultiArray	Raw controller state encoded as msgpack. Contains button, trigger, thumbstick, and pose data for each controller.
tf	tf2_msgs/TFMessage	Poses of the wrists and head with respect to the world frame world_frame.