Tutorial for Multi-Object Pick and Place from Isaac Cloud Service

Overview

This tutorial walks through the process of triggering pick and place workflow from Isaac cloud service using Mission Dispatch APIs. In this tutorial, you will setup Mission Dispatch on one x86_64 machine, launch mission client along with other ROS 2 nodes on the robot and trigger pick and place workflow by cloud APIs.

Prerequisites

• Complete the pick and place tutorial for pick and place using ROS 2 action servers first.

• If you are using Isaac Sim, complete the setup steps in the Isaac Sim tutorials, then follow the pick and place tutorial for Isaac Sim:

  1. Set Up Development Environment

  2. Build the Code

  3. Set Up Perception Deep Learning Models

• If you have a RealSense manipulator, complete all sections except “Run Pick and Place with Isaac Sim Platform.”

• Follow the setup instructions in Setup Hardware and Software for Real Robot with Isaac ROS Manipulation.

Tutorial Walkthrough

Setup Mission Dispatch

1. Start MQTT broker on the x86_64 machine.

   The MQTT broker is used for communication between the Mission Dispatch and the robots. There are many ways to run an MQTT broker, including as a system daemon, a standalone application, or a Docker container. The following example uses mosquitto as the MQTT broker.

   1. Create a file ~/mosquitto.sh with the following contents outside the container:

   CONFIG_FILE=/mosquitto.conf
   if [ $# != 2 ] ; then
       echo "usage: $0 <tcp_port> <websocket_port>"
       exit 1
   fi
   PORT=$1
   PORT_WEBSOCKET=$2
   echo "allow_anonymous true" >> $CONFIG_FILE
   echo "listener $PORT 0.0.0.0" >> $CONFIG_FILE
   echo "listener $PORT_WEBSOCKET" >> $CONFIG_FILE
   echo "protocol websockets" >> $CONFIG_FILE
   mosquitto -c $CONFIG_FILE
   

   2. Run the command outside the container:

   docker run -it --network host -v ~/mosquitto.sh:/mosquitto.sh -d eclipse-mosquitto:latest sh mosquitto.sh 1883 9001
   

2. Start Mission Dispatch with Docker.

   On the Postgres DatabaseLaunch the Mission Database Microservice

   1. Set the following environment variable:

   export POSTGRES_PASSWORD=<Any password>
   

   2. Start the Postgres database by running the following:

      docker run --rm --name postgres \
         --network host \
         -p 5432:5432 \
         -e POSTGRES_USER=postgres \
         -e POSTGRES_PASSWORD \
         -e POSTGRES_DB=mission \
        -d postgres:14.5
      

   3. Start the API and database server with the official Docker container.

   x86_64 (amd64)

   docker run -it --network host nvcr.io/nvidia/isaac/mission-database:4.3.0-amd64
   

   ARM64

   docker run -it --network host nvcr.io/nvidia/isaac/mission-database:4.3.0-arm64
   

   4. Start the mission dispatch server with the official docker container.

   x86_64 (amd64)

   docker run -it --network host nvcr.io/nvidia/isaac/mission-dispatch:4.3.0-amd64
   

   ARM64

   docker run -it --network host nvcr.io/nvidia/isaac/mission-dispatch:4.3.0-arm64
   

   Read this tutorial for more deployment options for Mission Dispatch.

3. Open http://localhost:5000/docs in a web browser.

Launch ROS 2 Nodes

Complete the following steps on the robot.

1. Complete Set Up Development Environment and Build isaac_ros_mission_client to build the packages.

2. Run the following launch files within the container to spin up mission_client:

   ros2 launch isaac_ros_vda5050_client_bringup isaac_ros_vda5050_client.launch.py \
       robot_type:=MANIPULATOR \
       serial_number:=arm01 \
       mqtt_host_name:=<mission_dispatch_ip>
   

3. Complete the required sections in the pick and place tutorial or the pick and place tutorial for Isaac Sim first, as described in the overview section above.

4. Run the following command to verify that the Isaac ROS Manipulation is working correctly:

   export ENABLE_MANIPULATOR_TESTING=on_robot  # use isaac_sim if running on Isaac Sim
   export ISAAC_ROS_MANIPULATION_TEST_CONFIG=<config_file_path_for_your_robot>
   python3 -m pytest ${ISAAC_ROS_WS}/src/isaac_ros_manipulation/isaac_ros_manipulation_bringup/test
   

   Note

   You can also use colcon test --packages-select isaac_ros_manipulation_bringup to run the tests. However, pytest has a better output and makes it easier to view status and progress of the tests. If these tests fail, refer to Isaac ROS Manipulation Testing Guide for more information. It is recommended to run the tests manually using launch_test and then manually inspecting the results.

5. Launch pick and place servers:

   On RealSense RobotIn Isaac Sim

   1. Open a new terminal inside the container and run:

   cd ${ISAAC_ROS_WS} && \
      source install/setup.bash
   

   1. Change the camera_type to REALSENSE in the configuration file before launching this workflow.

   ros2 launch isaac_ros_manipulation_bringup workflows.launch.py \
       manipulator_workflow_config:=$(ros2 pkg prefix --share isaac_ros_manipulation_bringup)/params/ur5e_robotiq_85_mac_and_cheese.yaml
   

   Note

   This tutorial was validated using ur_type: ur5e, ur_type: ur10e with gripper types robotiq_2f_140 and robotiq_2f_85. These values may be changed in the workflow configuration file to select between supported robots and grippers.

Trigger Pick and Place Workflow

1. Open http://localhost:5000/docs in a web browser.

2. Use the POST /robot endpoint to create robot with name arm01.

   {
       "labels": [],
       "heartbeat_timeout": 30,
       "name": "arm01"
   }
   

   When using the interactive documentation page, the default value for the the robot object name in the spec is ‘string’, you must change it from ‘string’ to another name that has more meaning, like ‘arm01’. Delete the prefix entry as shown in the video.

3. Trigger pick and place using POST /mission endpoint.

   {
       "robot": "arm01",
       "mission_tree": [
           {
               "name": "pick_place",
               "parent": "root",
               "action": {
                   "action_type": "multi_object_pick_and_place",
                   "action_parameters": {
                       "mode": "SINGLE_BIN",
                       "class_ids": "",
                       "target_poses": "{\"frame_id\": \"base_link\", \"poses\": [{\"position\": {\"x\": -0.25, \"y\": 0.45, \"z\": 0.50}, \"orientation\": {\"x\": -0.678, \"y\": 0.735, \"z\": 0.021, \"w\": 0.018}}]}"
                   }
               }
           }
       ],
       "timeout": 300,
       "needs_canceled": false
   }
   

   Note

   mode can be SINGLE_BIN or MULTI_BIN. class_ids is a string of comma separated class IDs. target_poses is a string serialized JSON of geometry_msgs/PoseArray.