`isaac_ros_teleop`#

Source code available on GitHub.

Quickstart#

Prerequisites#

This quickstart has been tested and qualified on Jetson AGX Thor and x86_64.

PICO 4 Ultra or Meta Quest 3 headset
If no XR headset is available, the emulator provided by Isaac Teleop Core can be used instead
A stable network connection meeting the CloudXR network requirements

Set Up Development Environment#

Set up your development environment by following the instructions in getting started.
(Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

Note

We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the development environment during installation, which will interrupt the quickstart process.

Build `isaac_ros_teleop`#

Install and build isaac_ros_teleop:

Activate the Isaac ROS environment:
```
isaac-ros activate
```

Install the prebuilt Debian package:

sudo apt-get update

sudo apt-get install -y ros-jazzy-isaac-ros-teleop

Install Git LFS:

sudo apt-get install -y git-lfs && git lfs install

Clone this repository under ${ISAAC_ROS_WS}/src:

cd ${ISAAC_ROS_WS}/src && \
   git clone --recurse-submodules -b release-4.5 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_teleop.git isaac_ros_teleop

Activate the Isaac ROS environment:
```
isaac-ros activate
```

Use rosdep to install the package’s dependencies:

sudo apt-get update

rosdep update && rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_teleop/isaac_ros_teleop --ignore-src -y

Build the package from source:

cd ${ISAAC_ROS_WS}/ && \
   colcon build --symlink-install --packages-up-to isaac_ros_teleop --base-paths ${ISAAC_ROS_WS}/src/isaac_ros_teleop/isaac_ros_teleop

Source the ROS workspace:

Note

Repeat this step in every terminal created inside the Isaac ROS environment. Because you built this package from source, you must source the enclosing workspace for ROS to find the package’s contents.
```
source install/setup.bash
```

Run CloudXR Server#

Whitelist firewall ports for CloudXR.

CloudXR requires certain network ports to be open. Depending on your firewall configuration, you may need to open them manually. For Quest and PICO headsets (WebXR Client), at the minimum, the CloudXR runtime and WebSocket Secure proxy ports must be open. If ufw is enabled, run:
```
sudo ufw allow 47998/udp
sudo ufw allow 49100,48322/tcp
```
If you are running the WebXR client from source and ufw is enabled, also open the web server ports:
```
sudo ufw allow 8080,8443/tcp
```
Start the CloudXR runtime. Be sure to review and accept the EULA:
```
python3 -m isaacteleop.cloudxr
```
Tip

To accept the EULA prompt in non-interactive settings, pass the flag:
```
python3 -m isaacteleop.cloudxr --accept-eula
```
In a new terminal, activate the Isaac ROS environment:
```
isaac-ros activate
```
Activate the CloudXR environment:
```
source ~/.cloudxr/run/cloudxr.env
```
Connect the XR headset to the teleop server. In your headset’s browser, open the CloudXR web client and follow the headset connection guide.

Note

If the CloudXR runtime is running on Thor, make sure to set the Video Codec to H.264, otherwise the headset will fail to connect.

Warning

The world frame of the headset is defined as the position of the headset and controllers at the moment of connection. Stand still and face the robot before connecting to establish a consistent world frame. To reset the world frame, disconnect and reconnect the headset while stationary.

Run Launch File#

Run the following launch file to spin up a demo of this package:
```
ros2 launch isaac_ros_teleop isaac_ros_teleop.launch.py
```

Visualize Results#

Open a new terminal and activate the Isaac ROS environment:
```
isaac-ros activate
```

Install RViz:

sudo apt install -y ros-jazzy-rviz2
source /opt/ros/jazzy/setup.bash

Open RViz:
```
rviz2
```
Then set the Fixed Frame to world. Then click on the Add button, select By display type, and choose TF under rviz_default_plugins.
With the controllers in your hands, start moving them. You should see the frames track your movements:

Note

The frames are converted to match ROS frame convention. Therefore, the poses are re-oriented from Right-Up-Back orientation to Forward-Left-Up for the x, y and z axes respectively.

Controller Reference#

The PICO 4 Ultra headset and Meta Quest 3 include two handheld controllers. The following table summarizes what each input does during teleoperation:

Input	Action
Left joystick	Move the robot: up = forward, down = backward, left = strafe left, right = strafe right
Right joystick — left / right	Rotate the robot in place (yaw)
Controller motion (6-DOF)	The end-effector pose tracks the physical controller; moving and rotating the controller moves the robot’s hand correspondingly
Triggers (each controller has two)	Open and close the finger joints of the tri-finger hand

Troubleshooting#

Isaac ROS Troubleshooting#

For solutions to problems with Isaac ROS, see troubleshooting.

API#

Usage#

ros2 launch isaac_ros_teleop isaac_ros_teleop.launch.py

ROS Parameters#

Launch Argument	Type	Default	Description
`ee_pose_topic`	`string`	`xr_teleop/ee_poses`	Topic name for published end-effector poses (`geometry_msgs/PoseArray`).
`root_twist_topic`	`string`	`xr_teleop/root_twist`	Topic name for published root velocity command (`geometry_msgs/TwistStamped`).
`root_pose_topic`	`string`	`xr_teleop/root_pose`	Topic name for published root pose command (`geometry_msgs/PoseStamped`).
`finger_joints_topic`	`string`	`xr_teleop/finger_joints`	Topic name for the retargeted finger joints (`sensor_msgs/JointState`).
`controller_data_topic`	`string`	`xr_teleop/controller_data`	Topic name for raw msgpack-encoded controller state (`std_msgs/ByteMultiArray`)
`rate_hz`	`double`	`60.0`	The publishing rate in Hz.
`world_frame`	`string`	`world`	World frame for message headers and TF parent frame.
`right_wrist_frame`	`string`	`right_wrist`	TF child frame name for the right wrist.
`left_wrist_frame`	`string`	`left_wrist`	TF child frame name for the left wrist.
`left_finger_joint_names`	`string`	`["left_hand_thumb_0_joint","left_hand_thumb_1_joint","left_hand_thumb_2_joint","left_hand_index_0_joint","left_hand_index_1_joint","left_hand_middle_0_joint","left_hand_middle_1_joint"]`	Left-hand finger joint names in TriHand retargeter order: thumb_rotation, thumb_proximal, thumb_distal, index_proximal, index_distal, middle_proximal, middle_distal.
`right_finger_joint_names`	`string`	`["right_hand_thumb_0_joint","right_hand_thumb_1_joint","right_hand_thumb_2_joint","right_hand_index_0_joint","right_hand_index_1_joint","right_hand_middle_0_joint","right_hand_middle_1_joint"]`	Right-hand finger joint names in TriHand retargeter order: thumb_rotation, thumb_proximal, thumb_distal, index_proximal, index_distal, middle_proximal, middle_distal.
`transform_rotation`	`string`	`[0.5, -0.5, -0.5, 0.5]`	Quaternion `[qx, qy, qz, qw]` to rotate XR poses into the ROS world frame. Defaults to the Right-Up-Back orientation to Forward-Left-Up.
`transform_translation`	`string`	`[0.0, 0.0, 0.0]`	Translation `[x, y, z]` applied to XR poses after rotation.

ROS Topics Published#

ROS Topic	Interface	Description
`ee_pose_topic`	geometry_msgs/PoseArray	The poses of the wrists or end-effectors, given in order of right wrist and then left wrist.
`root_twist_topic`	geometry_msgs/TwistStamped	The root twist command.
`root_pose_topic`	geometry_msgs/PoseStamped	The root pose command.
`finger_joints_topic`	sensor_msgs/JointState	The retargeted finger joint angles for the robot.
`controller_data_topic`	std_msgs/ByteMultiArray	Raw controller state encoded as msgpack. Contains button, trigger, thumbstick, and pose data for each controller.
`tf`	tf2_msgs/TFMessage	Poses of the wrists with respect to the world frame `world_frame`.

isaac_ros_teleop#