isaac_manipulator_servers

Source code on GitHub.

Overview

isaac_manipulator_servers package contains the implementation of the following servers:

• Object Detection Server: For detecting objects in the scene using an object detection network, such as RT-DETR.

• Pose Estimation Server: For estimating the pose of the object in the scene using a pose estimation network, such as FoundationPose.

• Object Info Server:, which wraps the aforementioned servers. This provides a common interface for getting object information, such as the object’s pose or the 2D bounding box of the object within an image.

From a high level, a Object Detection Server and Pose Estimation Server continuously subscribes to the inputs and stores the latest received input. For example, the Object Detection Server continuously subscribes to images and stores the latest image. Once the action is triggered, it will publish the latest image that it received to the specified topic. It then expects a pipeline, such as the RT-DETR pipeline, to subscribe this image and process it to produce a detection. This detection should then be published so that the Object Detection Server can receive it and pass it forward to the subsequent part of the pipeline.

API

ObjectDetectionServer

ROS Parameters

ROS Parameter	Type	Default	Description
action_name	string	object_detection	The name of the object detection action.
input_img_topic_name	string	image_color	The topic that the server will subscribe to for input images.
output_img_topic_name	string	object_detection_server/image_color	When executed, the server will publish messages to this topic.
input_detections_topic_name	string	detections_output	The server receives the actual object detection through this topic.
output_detections_topic_name	string	object_detection_server/detections_output	The server publishes the received input detections to this topic.
sub_qos	string	SENSOR_DATA	Subscription QoS value.
pub_qos	string	SENSOR_DATA	Publisher QoS value.

ROS Topics Subscribed

ROS Topic	Interface	Description
<input_img_topic_name>	sensor_msgs::msg::Image	The input image that will be used for object detection. The latest image is stored, and will be sent through the <output_img_topic_name> when the action is triggered.
<input_detections_topic_name>	vision_msgs::msg::Detection2DArray	The resultant detections coming from a object detection model.

ROS Topics Published

ROS Topic	Interface	Description
<output_img_topic_name>	sensor_msgs::msg::Image	The image that is forwarded to object detection model
<output_detections_topic_name>	vision_msgs::msg::Detection2DArray	Detections forwarded from object detection model.

ROS Actions Advertised

ROS Action	Interface	Description
<action_name>	isaac_manipulator_interfaces::action::DetectObjects	Action server for object detection models.

DopeServer

Note

At the present, this node is not used.

ROS Parameters

ROS Parameter	Type	Default	Description
action_name	string	dope	The name of the DOPE action server.
in_img_topic_name	string	image_color	The topic that the server will subscribe to for input images.
out_img_topic_name	string	dope_server/image_color	When executed, the server will publish an image to this topic.
in_camera_info_topic_name	string	camera_info	The topic that the server will subscribe to for input camera info messages.
out_camera_info_topic_name	string	dope_server/camera_info	When executed, the server will publish a camera info to this topic.
in_pose_estimate_topic_name	string	poses	The server receives the actual poses through this topic.
out_pose_estimate_topic_name	string	dope_server/poses	The server publishes the received input poses to this topic.
in_detections_topic_name	string	detections	The server receives the actual 2D detections through this topic.
out_detections_topic_name	string	dope_server/detections	The server publishes the received input 2D detections to this topic.
enable_2d_detections	bool	false	Whether or not the server should expect to receive 2D detections or not.
sub_qos	string	SENSOR_DATA	Subscription QoS value.
pub_qos	string	SENSOR_DATA	Publisher QoS value.

ROS Topics Subscribed

ROS Topic	Interface	Description
<in_img_topic_name>	sensor_msgs::msg::Image	The input image that is used for pose estimation. The latest image is stored.
<in_camera_info_topic_name>	sensor_msgs::msg::CameraInfo	CameraInfo messages for input images. The latest CameraInfo is stored.
<in_pose_estimate_topic_name>	vision_msgs::msg::Detection3DArray	The resultant pose coming from the model.
<in_detections_topic_name>	vision_msgs::msg::Detection2DArray	The resultant 2D detections coming from the model.

ROS Topics Published

ROS Topic	Interface	Description
<out_img_topic_name>	sensor_msgs::msg::Image	The image that is forwarded to the model
<out_camera_info_topic_name>	sensor_msgs::msg::CameraInfo	CameraInfo messages which are forwarded to the model.
<out_pose_estimate_topic_name>	vision_msgs::msg::Detection3DArray	Pose estimation forwarded from the model.
<out_detections_topic_name>	vision_msgs::msg::Detection2DArray	2D detections forwarded from the model.

ROS Actions Advertised

ROS Action	Interface	Description
<action_name>	isaac_manipulator_interfaces::action::EstimatePoseDope	Action server for DOPE DNN model.

FoundationPoseServer

ROS Parameters

ROS Parameter	Type	Default	Description
action_name	string	estimate_pose_foundation_pose	The name of the foundation pose action.
in_img_topic_name	string	image_color	The topic that the server will subscribe to for input images.
out_img_topic_name	string	foundation_pose_server/image_color	When executed, the server will publish images to this topic.
in_camera_info_topic_name	string	camera_info	The topic that the server will subscribe to for input camera info messages.
out_camera_info_topic_name	string	foundation_pose_server/camera_info	When executed, the server will publish camera info messages to this topic.
in_depth_topic_name	string	depth	The topic that the server will subscribe to for input depth images.
out_depth_topic_name	string	foundation_pose_server/depth	When executed, the server will publish depth images to this topic.
in_pose_estimate_topic_name	string	poses	The server receives the actual pose estimation through this topic.
out_pose_estimate_topic_name	string	foundation_pose_server/poses	The server publishes the received pose estimations to this topic.
out_bbox_topic_name	string	foundation_pose_server/bbox	The server publishes the bounding box that it receives as part of the goal to this topic.
sub_qos	string	SENSOR_DATA	Subscription QoS value.
pub_qos	string	SENSOR_DATA	Publisher QoS value.

ROS Topics Subscribed

ROS Topic	Interface	Description
<in_img_topic_name>	sensor_msgs::msg::Image	The input image that is used for pose estimation. The latest image is stored.
<in_camera_info_topic_name>	sensor_msgs::msg::CameraInfo	CameraInfo messages for input images. The latest camera info is stored.
<in_depth_topic_name>	sensor_msgs::msg::Image	The input depth image that is used for pose estimation. The latest depth image is stored.
<in_pose_estimate_topic_name>	vision_msgs::msg::Detection3DArray	The resultant pose estimation coming from the model.

ROS Topics Published

ROS Topic	Interface	Description
<out_img_topic_name>	sensor_msgs::msg::Image	The image that is forwarded to the model.
<out_camera_info_topic_name>	sensor_msgs::msg::CameraInfo	CameraInfo messages which are forwarded to the model.
<out_depth_topic_name>	sensor_msgs::msg::Image	The depth image that is forwarded to the model
<out_bbox_topic_name>	vision_msgs::msg::Detection2DArray	Bounding box data forwarded to the model.
<out_detections_topic_name>	vision_msgs::msg::Detection3DArray	Pose estimation forwarded from the model.

ROS Actions Advertised

ROS Action	Interface	Description
<action_name>	isaac_manipulator_interfaces::action::EstimatePoseFoundationPose	Action server for Foundation Pose DNN model.

ObjectInfoServer

ROS Parameters

ROS Parameter	Type	Default	Description
object_detection_backend	string	RT_DETR	Object detection backend to use. The options currently supported are RT_DETR and DOPE``.
pose_estimation_backend	string	FOUNDATION_POSE	Pose estimation backend to use. The options currently supported are FOUNDATION_POSE and DOPE.

ROS Actions Advertised

ROS Action	Interface	Description
/get_objects	isaac_manipulator_interfaces::action::GetObjects	Wrapper action server for object detection server.
/get_object_pose	isaac_manipulator_interfaces::action::GetObjectPose	Wrapper action server for pose estimation server.

ROS Services Advertised

ROS Service	Interface	Description
/clear_objects	isaac_manipulator_interfaces::srv::ClearObjects	Service to clear the cached object info like 2D detections and pose estimation.