FoundationPose Tracking

Overview

This tutorial walks you through a graph to run FoundationPose tracking using FoundationPose FoundationPose tracking has similar functionality as FoundationPose pose estimation. However, it utilizes the pose from the previous frame and requires only the refine network without batching which can significantly increase the speed of the pipeline. Therefore, you can use the FoundationPose pose estimation to obtain the initial pose on the first frame and then switch to FoundationPose tracking for subsequent frames.

Note

To ensure the tracking pipeline is able to get the initial pose estimation, play the rosbag after the launch file is fully started.

Tutorial Walkthrough

1. Complete the Isaac ROS FoundationPose Quickstart Guide.

2. Download tracking rosbag from NGC:

NVIDIA Internal:

The NGC CLI must be set up once per system.

1. Install the NGC CLI from here.

   Note

   Jetson Only: Do not edit the ~/.bash_profile file as suggested by the official NGC CLI instructions.

   Instead, add the NGC CLI to your PATH through ~/.bashrc:

   s="export PATH=\"\$PATH:$(pwd)/ngc-cli\""
   f="$HOME/.bashrc"; grep -qxF "$s" $f || echo "$s" | tee -a $f && source $f
   

2. Generate an API key here.

3. Run ngc config set:

   When prompted, enter the API key you generated earlier.

   ngc config set
   

4. To confirm that the NGC CLI is set up correctly, run the following command:

   grep -E "^apikey\s*=\s*" ~/.ngc/config
   

   You should see an output with a valid API key:

   apikey = <your_api_key>
   

Make sure required libraries are installed.

sudo apt-get install -y curl jq tar

Then, run these commands to download the asset from NGC:

NGC_ORG="nvstaging"
NGC_TEAM="isaac"
PACKAGE_NAME="isaac_ros_foundationpose"
NGC_RESOURCE="isaac_ros_foundationpose_assets"
NGC_FILENAME="foundationpose_tracking.tar.gz"
MAJOR_VERSION=3
MINOR_VERSION=3
NGC_CONFIG_FILE="$HOME/.ngc/config"
API_KEY=$(grep -E "^apikey\s*=\s*" $NGC_CONFIG_FILE | \
    awk -F "=" '{print $2}' | tr -d '[:space:]')
TOKEN=$(curl -s -u "\$oauthtoken":"$API_KEY" -H 'Accept:application/json' \
    "https://authn.nvidia.com/token?service=ngc"\
"&scope=group/ngc:$NGC_ORG&group/ngc:$NGC_ORG/$NGC_TEAM" | \
    jq -r '.token')
VERSION_REQ_URL="https://api.ngc.nvidia.com/v2/org/$NGC_ORG/team/$NGC_TEAM/resources/$NGC_RESOURCE/versions"
AVAILABLE_VERSIONS=$(curl -s -H "Authorization: Bearer ${TOKEN}" \
    -H "Accept: application/json" "$VERSION_REQ_URL")
LATEST_VERSION_ID=$(echo $AVAILABLE_VERSIONS | jq -r "
    .recipeVersions[]
    | .versionId as \$v
    | \$v | select(test(\"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+$\"))
    | split(\".\") | {major: .[0]|tonumber, minor: .[1]|tonumber, patch: .[2]|tonumber}
    | select(.major == $MAJOR_VERSION and .minor <= $MINOR_VERSION)
    | \$v
    " | sort -V | tail -n 1
)
if [ -z "$LATEST_VERSION_ID" ]; then
    echo "No corresponding version found for Isaac ROS $MAJOR_VERSION.$MINOR_VERSION"
    echo "Found versions:"
    echo $AVAILABLE_VERSIONS | jq -r '.recipeVersions[].versionId'
else
    mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets && \
    FILE_REQ_URL="https://api.ngc.nvidia.com/v2/org/$NGC_ORG/team/$NGC_TEAM/resources/\
$NGC_RESOURCE/versions/$LATEST_VERSION_ID/files/$NGC_FILENAME" && \
    curl -LO --request GET "${FILE_REQ_URL}" \
        -H "Authorization: Bearer ${TOKEN}" \
        -H "Content-Type: application/json" && \
    tar -xf ${NGC_FILENAME} -C ${ISAAC_ROS_WS}/isaac_ros_assets && \
    rm ${NGC_FILENAME}
fi

3. Open a new terminal and launch the Docker container using the run_dev.py script:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
     ./scripts/run_dev.py
   

RosbagRealSense CameraHawk Camera

4. Continuing inside the container, install the following dependencies:

   NVIDIA Internal: Run these commands to add the internal apt repository:

   sudo apt install curl -y
   k="/usr/share/keyrings/nvidia-isaac-ros.gpg"
   curl -fsSL https://isaac.download.nvidia.com/isaac-ros/repos.key | sudo gpg --dearmor | sudo tee -a $k > /dev/null
   f="/etc/apt/sources.list.d/nvidia-isaac-ros.list" && sudo touch $f
   s="deb [signed-by=$k] https://urm.nvidia.com/artifactory/sw-isaac-staging-debian-local jammy release-3.3"
   grep -qxF "$s" $f || echo "$s" | sudo tee -a $f
   
   pin_content=$'package: *\nPin: origin isaac.download.nvidia.com\nPin-Priority: 400'
   echo "$pin_content" | sudo tee /etc/apt/preferences.d/isaac-ros
   
   sudo apt-get update
   

   sudo apt-get install -y ros-humble-isaac-ros-examples
   

5. Run the following launch files to spin up a demo of this package:

   Launch isaac_ros_foundationpose:

   ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=foundationpose_tracking interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/quickstart_interface_specs.json mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mustard/textured_simple.obj score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan
   

   Then open another terminal, and enter the Docker container again:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
      ./scripts/run_dev.py
   

   Then, play the rosbag:

   ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/foundationpose_tracking.bag/
   

Visualize Results

1. Open a new terminal inside the Docker container:

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
      ./scripts/run_dev.py
   

   RosbagRealSense CameraHawk Camera

   2. launch RViz2 to visualize the output

      rviz2 -d  $(ros2 pkg prefix isaac_ros_foundationpose --share)/rviz/foundationpose_tracking.rviz
      

   3. You should see a RViz2 window open as shown below showing the 3D bounding box overlaid over the input image