This tutorial walks through running whole-body XR teleoperation on the Unitree G1
humanoid robot. The application combines AGILE locomotion, bimanual inverse kinematics,
and finger control, all driven by an XR headset.
You first run the application in MuJoCo simulation, then deploy on real hardware.
Set up your development environment by following the instructions in getting started.
(Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.
Note
We strongly recommend installing all sensor dependencies before starting any quickstarts.
Some sensor dependencies require restarting the development environment during installation, which will interrupt the quickstart process.
Set up the Intel RealSense camera by following the
RealSense setup tutorial.
For Docker environments, the RealSense setup tutorial adds the realsense
Docker image key and rebuilds the Isaac ROS environment with
isaac-rosactivate--build-local.
Set up the Unitree G1 with the host machine connected to it by following these guides:
Repeat this step in every terminal created inside the Isaac ROS environment. Because you built this package from source, you must source the enclosing workspace for ROS to find the package’s contents.
CloudXR requires certain network ports to be open. Depending on your firewall configuration, you may need to open them manually.
For Quest and PICO headsets (WebXR Client), at the minimum, the CloudXR runtime and WebSocket Secure proxy ports must be open. If ufw is enabled, run:
sudoufwallow47998/udp
sudoufwallow49100,48322/tcp
If you are running the WebXR client from source and ufw is enabled, also open the web server ports:
This opens the MuJoCo viewer with the G1 robot. The virtual gantry holds the robot
upright during startup. Press G to toggle the gantry on/off, and use [ / ]
to shorten or lengthen the rope. It is recommended to disable the gantry once the robot
is stable. The robot may oscillate at startup and will only stabilize once the controller
is active.
Note
blend_ratio defaults to 0.0. After the controller is active,
enable the policy with
ros2paramset/safety_controllerblend_ratio1.0.
If the CloudXR runtime is running on Jetson, make sure to set the VideoCodec to H.264,
otherwise the headset will fail to connect.
Warning
The world frame of the headset is defined as the position of the headset and controllers
at the moment of connection. Stand still and face the robot before connecting to establish
a consistent world frame. To reset the world frame, disconnect and reconnect the headset
while stationary.
With the controllers in your hands, start moving them. Verify that the robot’s arms
track your movements in the MuJoCo viewer.
Warning
Before operating on real hardware:
Ensure the working area is free of any persons or other potential hazards.
Always start with blend_ratio at 0.0. You can increase from 0.0 to 1.0 in a single step since the ratio is smoothed internally.
Ensure the waist yaw joint is close to zero before launching. The controller does not actuate it and holds it at its current position, so a rotated torso can degrade balance.
Have the disable command ready (refer to the disable step below).
Set up the network. Clone isaac_ros_robots and run the setup script outside the docker container on the host machine:
The world frame of the headset is defined as the position of the headset and controllers
at the moment of connection. Stand still and face the robot before connecting to establish
a consistent world frame. To reset the world frame, disconnect and reconnect the headset
while stationary.
To disable the robot, set the blend ratio back to zero:
ros2paramset/safety_controllerblend_ratio0.0
Keep this command in your shell history so you can execute it quickly if something goes wrong.
Enable the robot by setting the blend ratio:
ros2paramset/safety_controllerblend_ratio1.0
The robot starts tracking your hand movements.
Note
After several minutes of operation, the G1 hands may lower due to temperature limits. Allow the robot to cool down before resuming
Warning
When operating real robot hardware, stop the robot (set blend_ratio to
0.0) before disconnecting the teleop application.
If the XR headset disconnects at any point during teleoperation, the G1’s hands hold their last commanded position.
ros2_control_node Times Out Waiting for the Robot State#
On real hardware, ros2_control_node may fail to activate the G1 hardware
interface with output similar to the following:
[ros2_control_node-1] [ERROR] [UnitreeG1SystemInterface]: Timeout waiting for robot state
[ros2_control_node-1] [ERROR] [resource_manager]: Failed to 'activate' hardware 'UnitreeG1System'
[ros2_control_node-1] terminate called after throwing an instance of 'std::runtime_error'
[ros2_control_node-1] what(): Failed to set the initial state of the component : UnitreeG1System to active
This error indicates that the host did not receive the initial robot state from
the G1. Recover the connection as follows:
Stop all Isaac ROS containers.
Reboot both the G1 and the host computer (for example, Jetson AGX Thor).
On the host, rerun the network setup script and select the network interface
that is physically connected to the G1:
Relaunch the workflow. If the timeout persists, stop all containers and run
setup_network.py again. Establishing the connection might require more
than one attempt.
Test Without an XR Headset (Interactive Markers Mode)#
If the XR headset is unavailable or you want to isolate whether an issue is
with XR or the robot itself, launch with input_mode:=markers:
RViz opens automatically with six-degrees-of-freedom (6-DOF) interactive markers for each wrist.
The /ik_controller/reference_pose topic
replaces the /xr_teleop/ee_poses topic in this mode.
In the RViz Displays panel, find the IK Target Marker display and set
its Interactive Markers Namespace to /ik_controller_marker. You can
then drag the wrist markers to command the arms.
