Tutorial for Pick and Place from Isaac Cloud Service
Overview
This tutorial walks through the process of triggering pick and place workflow from Isaac cloud service using Mission Dispatch APIs. In this tutorial, you will setup Mission Dispatch on one x86_64 machine, launch mission client along with other ROS2 nodes on the robot and trigger pick and place workflow by cloud APIs.
Prerequisites
Complete the pick and place tutorial for pick and place using ROS2 action servers first.
If you are using Isaac Sim, complete the following sections in the pick and place tutorial for Isaac Sim:
Set Up Development Environment
Build the Code
Set Up Perception Deep Learning Models
If you have a RealSense manipulator, complete all sections except “Run Pick and Place with Isaac Sim Platform.”
Tutorial Walkthrough
Setup Mission Dispatch
The following steps should be done on an x86_64 machine.
Start MQTT broker on the x86_64 machine.
The MQTT broker is used for communication between the Mission Dispatch and the robots. There are many ways to run an MQTT broker, including as a system daemon, a standalone application, or a Docker container. The following example uses mosquitto as the MQTT broker.
Create a file
~/mosquitto.sh
with the following contents outside the container:
CONFIG_FILE=/mosquitto.conf if [ $# != 2 ] ; then echo "usage: $0 <tcp_port> <websocket_port>" exit 1 fi PORT=$1 PORT_WEBSOCKET=$2 echo "allow_anonymous true" >> $CONFIG_FILE echo "listener $PORT 0.0.0.0" >> $CONFIG_FILE echo "listener $PORT_WEBSOCKET" >> $CONFIG_FILE echo "protocol websockets" >> $CONFIG_FILE mosquitto -c $CONFIG_FILE
Run the command outside the container:
docker run -it --network host -v ~/mosquitto.sh:/mosquitto.sh -d eclipse-mosquitto:latest sh mosquitto.sh 1883 9001
Start Mission Dispatch with Docker.
Set the following environment variable:
export POSTGRES_PASSWORD=<Any password>
Start the Postgres database by running the following:
docker run --rm --name postgres \ --network host \ -p 5432:5432 \ -e POSTGRES_USER=postgres \ -e POSTGRES_PASSWORD \ -e POSTGRES_DB=mission \ -d postgres:14.5
Start the API and database server with the official Docker container.
docker run -it --network host nvcr.io/nvidia/isaac/mission-database:3.2.0
To see what the configuration options are, add
--help
option after the command:docker run -it --network host nvcr.io/nvidia/isaac/mission-database:3.2.0 --help
For example, if you want to change the port for the user API from the default 5000 to 5002, add
--port 5002
configuration option in the command.Read this tutorial for more deployment options for Mission Dispatch.
Open
http://localhost:5000/docs
in a web browser.
Launch ROS2 Nodes
Complete the following steps on the robot.
Complete Set Up Development Environment and Build isaac_ros_mission_client to build the packages.
Run the following launch files within the container to spin up
mission_client
:ros2 launch isaac_ros_vda5050_nav2_client_bringup isaac_ros_vda5050_client.launch.py \ robot_type:=arm \ launch_ws_bridge:=True \ serial_number:=arm01 \ mqtt_host_name:=<mission_dispatch_ip>
Complete the required sections in the pick and place tutorial or the pick and place tutorial for Isaac Sim first as described in Overview.
Launch pick and place servers:
Inside the container, launch the tool_communication.py script:
ros2 run ur_robot_driver tool_communication.py --ros-args -p robot_ip:=<ROBOT_IP_ADDRESS>
Open a new terminal inside the container and run:
cd ${ISAAC_ROS_WS} && \ source install/setup.bash
ros2 launch isaac_manipulator_pick_and_place ur_pick_and_place.launch.py \ ur_type:=<UR_TYPE> robot_ip:=<ROBOT_IP_ADDRESS> \ gripper_type:=<GRIPPER_TYPE> camera_type:=hawk setup:=<SETUP_NAME> \ ess_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.1.0_onnx/ess.engine
ros2 launch isaac_manipulator_pick_and_place ur_pick_and_place.launch.py \ ur_type:=<UR_TYPE> robot_ip:=<ROBOT_IP_ADDRESS> \ gripper_type:=<GRIPPER_TYPE> camera_type:=realsense setup:=<SETUP_NAME>
This tutorial was validated using
ur_type:=ur5e
,ur_type:=ur10e
with gripper types ofrobotiq_2f_140
androbotiq_2f_85
.Open Isaac Sim and load the scene below and hit Play.
https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Samples/ROS2/Scenario/isaac_manipulator_ur10e_robotiq_2f_140.usd
#. Set this flag to disable SHM on Isaac Sim. Make sure that file exists in the Docker container before running it. Do this on all terminals to make sure that all DDS communication runs over UDP.
export FASTRTPS_DEFAULT_PROFILES_FILE=/usr/local/share/middleware_profiles/rtps_udp_profile.xml
Inside the container, run:
ros2 launch isaac_manipulator_pick_and_place isaac_sim_workflows.launch.py \ workflow_type:=pick_and_place
For increased reliability, we recommend using the
custom_mesh
orcuboid
approach:For
custom_mesh
, this can be done by settingobject_attachment_type:=custom_mesh
andattach_object_mesh_file_path:=<PATH-TO-MESH>
in the launch files above.For
cuboid
, this can be done by settingobject_attachment_type:=cuboid
and settingobject_attachment_scale:=<CHOSEN_SCALE>
in the launch files above. TheCHOSEN_SCALE
consists of 3 floating point values, representing the dimensions of the cuboid.
Warning
If you see the log
No depth images from X seconds
, consider changing thefilter_depth_buffer_time
to a higher value (the unit is seconds). This will allow object attachment to buffer more depth images from the past. The caveat is that the software will operate on an older set of data that might lead to creating object spheres that might not model the object position accurately.The other parameter of interest is the
time_sync_slop
parameter. It defines the synchronization threshold for finding a matched pair of depth images, joint states and transforms. For slower systems, slightly tweaking the value up will allow for collision sphere and obstacle avoidance to work. Using a very large value will synchronize older pieces of data together leading to unpredictable downstream effects in planning, collision voxel cloud generation and obstacle avoidance.This only applies when
SPHERE
is used as the object attachment type. The other modes do not use depth as input to create the object spheres.If the gripper is in a bad state, that is, it is moving in on it self and causing the robot movement to self collide, you need to open the gripper as Isaac Sim caches the last known joint state in it’s action graphs and pushes that in when you hit Play. If you need to close the gripper, just make position
0.623
.You must have the launch file running to run this command.
ros2 action send_goal /robotiq_gripper_controller/gripper_cmd \ control_msgs/action/GripperCommand "{command: {position: 0.0, max_effort: 0.0}}"
Trigger Pick and Place Workflow
Open
http://localhost:5000/docs
in a web browser.Use the
POST /robot
endpoint to create robot with namearm01
.{ "labels": [], "heartbeat_timeout": 30, "name": "arm01" }
When using the interactive documentation page, the default value for the the robot object
name
in thespec
is ‘string’, you must change it from ‘string’ to another name that has more meaning, like ‘arm01’. Delete theprefix
entry as shown in the video.Trigger object detection using
POST /mission
endpoint. Here is an example input:{ "robot": "arm01", "mission_tree": [ { "name": "get_objects", "parent": "root", "action": { "action_type": "get_objects", "action_parameters": {} } } ], "timeout": 300, "needs_canceled": false }
If the mission completes successfully, use
GET /detection_results
endpoint to get object detection results. The detection results containobject_id
andclass_id
, which are needed for pick and place workflow.If the
get_objects
mission fails orGET /detection_results
endpoint returns an empty list, repeat step 2 to send the mission again.Trigger pick and place using
POST /mission
endpoint.object_id
andclass_id
come from the detection results.place_pose
is a comma separated string with 7 floats, representing the position and quaternion of where the gripper drops the object in the manipulatorbase_link
frame. For the example scene in Isaac Sim, setplace_pose
as"-0.35,0.2,0.4,0.996,0.066,0.042,0.034"
to drop the object into the bin.{ "robot": "arm01", "mission_tree": [ { "name": "pick_place", "parent": "root", "action": { "action_type": "pick_and_place", "action_parameters": { "object_id": "<object_id>", "class_id": "<class_id>", "place_pose": "<position_x>, <position_y>, <position_z>, <Quaternion_x>, <Quaternion_y>, <Quaternion_z>, <Quaternion_w>" } } } ], "timeout": 300, "needs_canceled": false }
Note
If the
pick_and_place
mission fails, repeat steps 2-4 to try again.The pick and place server will cache object poses. If you rerun the pipeline and the object position is different, clear the cache using the
POST /mission
endpoint. Theobject_ids
parameter can be either empty to clear all objects or a comma separated string to clear specific objects.
{ "robot": "arm01", "mission_tree": [ { "name": "clear_objects", "parent": "root", "action": { "action_type": "clear_objects", "action_parameters": { "object_ids": "" } } } ], "timeout": 300, "needs_canceled": false }
The steps above shows how to trigger pick and place workflow using Mission Dispatch APIs. We provide a basic UI as an example showing how you can develop applications using our cloud APIs.
You can find the source code in
mission_dispatch/scripts/pick_and_place_ui.py
.To start the UI on your x86_64 machine:
Clone the code, navigate to the start script, and start the UI:
git clone --recurse-submodules https://github.com/nvidia-isaac/isaac_mission_dispatch.git cd mission_dispatch && scripts/run_dev.sh
Run the UI in the Docker container
bazel run //scripts:pick_and_place_ui -- http://localhost:5000 ws://<robot_ip>:9090 /resize/image
If you are using Isaac Sim,
robot_ip
must belocalhost
. If you are running on a RealSense manipulator, it is the IP address of your Jetson machine.
Enter the robot name
arm01
and click Run Object Detection to trigger object detection. The UI will show an image with bounding boxes for detected objects.Click on the image to select the object to pick and enter the place pose.
Click Submit to trigger pick and place pipeline.
To run clear objects mission, click Clear Objects.