Tutorial for Multi-Object Pick and Place from Isaac Cloud Service

Overview

This tutorial walks through the process of triggering pick and place workflow from Isaac cloud service using Mission Dispatch APIs. In this tutorial, you will setup Mission Dispatch on one x86_64 machine, launch mission client along with other ROS 2 nodes on the robot and trigger pick and place workflow by cloud APIs.

Prerequisites

• Complete the pick and place tutorial for pick and place using ROS 2 action servers first.

• If you are using Isaac Sim, complete the following sections in pick and place tutorial for Isaac Sim:

  1. Set Up Development Environment

  2. Build Isaac ROS cuMotion

  3. Launch Isaac Sim

• Follow the setup instructions in Setup Hardware and Software for Real Robot.

Tutorial Walkthrough

Setup Mission Dispatch

The following steps should be done on an x86_64 machine.

1. Start MQTT broker on the x86_64 machine.

   The MQTT broker is used for communication between the Mission Dispatch and the robots. There are many ways to run an MQTT broker, including as a system daemon, a standalone application, or a Docker container. The following example uses mosquitto as the MQTT broker.

   1. Create a file ~/mosquitto.sh with the following contents outside the container:

   CONFIG_FILE=/mosquitto.conf
   if [ $# != 2 ] ; then
       echo "usage: $0 <tcp_port> <websocket_port>"
       exit 1
   fi
   PORT=$1
   PORT_WEBSOCKET=$2
   echo "allow_anonymous true" >> $CONFIG_FILE
   echo "listener $PORT 0.0.0.0" >> $CONFIG_FILE
   echo "listener $PORT_WEBSOCKET" >> $CONFIG_FILE
   echo "protocol websockets" >> $CONFIG_FILE
   mosquitto -c $CONFIG_FILE
   

   2. Run the command outside the container:

   docker run -it --network host -v ~/mosquitto.sh:/mosquitto.sh -d eclipse-mosquitto:latest sh mosquitto.sh 1883 9001
   

2. Start Mission Dispatch with Docker.

   1. Set the following environment variable:

   export POSTGRES_PASSWORD=<Any password>
   

   2. Start the Postgres database by running the following:

   docker run --rm --name postgres \
       --network host \
       -p 5432:5432 \
       -e POSTGRES_USER=postgres \
       -e POSTGRES_PASSWORD \
       -e POSTGRES_DB=mission \
       -d postgres:14.5
   

   3. Start the API and database server with the official Docker container.

   docker run -it --network host nvcr.io/nvidia/isaac/mission-database:4.0.0
   

   4. Start the mission dispatch server with the official docker container.

   docker run -it --network host nvcr.io/nvidia/isaac/mission-dispatch:4.0.0
   

   Read this tutorial for more deployment options for Mission Dispatch.

3. Open http://localhost:5000/docs in a web browser.

Launch ROS 2 Nodes

Complete the following steps on the robot.

1. Complete Set Up Development Environment and Build isaac_ros_mission_client to build the packages.

2. Run the following launch files within the container to spin up mission_client:

   ros2 launch isaac_ros_vda5050_client_bringup isaac_ros_vda5050_client.launch.py \
       robot_type:=MANIPULATOR \
       launch_ws_bridge:=True \
       serial_number:=arm01 \
       mqtt_host_name:=<mission_dispatch_ip>
   

3. Complete the required sections in the pick and place tutorial or the pick and place tutorial for Isaac Sim first as described in Overview.

4. Run the following command to verify that Isaac for Manipulation is working correctly:

   export ENABLE_MANIPULATOR_TESTING=on_robot  # use isaac_sim if running on Isaac Sim
   export ISAAC_MANIPULATOR_TEST_CONFIG=<config_file_path_for_your_robot>
   python -m pytest ${ISAAC_ROS_WS}/src/isaac_manipulator/isaac_manipulator_bringup/test
   

   Note

   One can also use colcon test --packages-select isaac_manipulator_bringup to run the tests. However, pytest has a better output and makes it easier to view status and progress of the tests. If these tests fail, please look at Isaac for Manipulation Testing Guide for more information. It is recommended to run the tests manually using launch_test and then manually inspecting the results.

5. Launch pick and place servers:

   On RealSense RobotIn Isaac Sim

   1. Open a new terminal inside the container and run:

   cd ${ISAAC_ROS_WS} && \
      source install/setup.bash
   

   Realsense

   1. Change the camera_type to REALSENSE in the configuration file before launching this workflow.

   ros2 launch isaac_manipulator_bringup workflows.launch.py \
       manipulator_workflow_config:=$(ros2 pkg prefix --share isaac_manipulator_bringup)/params/ur5e_robotiq_85_mac_and_cheese.yaml
   

   Note

   This tutorial was validated using ur_type: ur5e, ur_type: ur10e with gripper types robotiq_2f_140 and robotiq_2f_85. These values may be changed in the workflow configuration file to select between supported robots and grippers.

Trigger Pick and Place Workflow

1. Open http://localhost:5000/docs in a web browser.

2. Use the POST /robot endpoint to create robot with name arm01.

   {
       "labels": [],
       "heartbeat_timeout": 30,
       "name": "arm01"
   }
   

   When using the interactive documentation page, the default value for the the robot object name in the spec is ‘string’, you must change it from ‘string’ to another name that has more meaning, like ‘arm01’. Delete the prefix entry.

3. Trigger pick and place using POST /mission endpoint.

   {
       "robot": "arm01",
       "mission_tree": [
           {
               "name": "pick_place",
               "parent": "root",
               "action": {
                   "action_type": "multi_object_pick_and_place",
                   "action_parameters": {
                       "mode": "SINGLE_BIN",
                       "class_ids": "",
                       "target_poses": "{\"frame_id\": \"base_link\", \"poses\": [{\"position\": {\"x\": -0.25, \"y\": 0.45, \"z\": 0.50}, \"orientation\": {\"x\": -0.678, \"y\": 0.735, \"z\": 0.021, \"w\": 0.018}}]}"
                   }
               }
           }
       ],
       "timeout": 300,
       "needs_canceled": false
   }
   

Note

mode can be SINGLE_BIN or MULTI_BIN. class_ids is a string of comma separated class IDs. target_poses is a string serialized JSON of geometry_msgs/PoseArray.