Tracebot In-Gripper

Context and methodology

This dataset was created to investigate pose refinement methods of transparent objects when held by a gripper. The aim is to  quantitatively evaluate the performance of various methods in that context. We collected diverse data for the investigation, including  glass and plastic objects, filled with liquid and empty, properties like opacity and index of refraction are varied. The selected objects also vary significantly in shape and size. Additionally, uniform and highly-textured backgrounds are tested.

Technical details

The dataset is collected by moving an object held by a robot gripper in front of two static cameras. The same gripper poses are collected for every scene, and the camera poses are obtained through inverse kinematics of the robot arm, after calibration. We use 3D-DAT (https://github.com/markus-suchi/3D-DAT/releases/tag/v1.0.0) for annotation, placing object models in the virtual 3D scene, and manually correcting their poses based on their reprojection error in the different RGB views.

To obtain 3D object models, the physical objects are coated using a mat spray paint after collecting the different scenes. A high-quality depth sensor (Photoneo MotionCam-3D scanner, https://www.photoneo.com/) is used to reconstruct them. The set of 15 objects used in our experiments is illustrated in Figure 3, and includes plastics and glass objects, filled or empty with a variety of shapes, and a variety of sizes.

A total of 22 scenes are collected using two cameras (leading to a total of 44 set of images). The cameras are Intel Realsense D435 (https://www.intelrealsense.com/depth-camera-d435i/), saving both the RGB image and the depth image at a resolution of 1280 × 720 pixels. The robotic arm performs a sequence of pose holding the object in various grasp pose, ensuring that the camera frustum is well covered while guaranteeing that mosst of the object is in view. The sequence consists in 16 poses. The light is uniform and comes from the top of the scene. All objects are collected in front of a uniform background and in front of a textured background.

The "objects/" folder contains the object models as well as a configuration file relating the model, its scale and its object id for annotation purposes. For each scene in the "scenes/" folder, the structure is as follow:

• rgb/ contains the color images
• depth/ contains the depth obtained with the Realsense D435 camera
• masks/ contains the groundtruth masks of the object
• groundtruth_handeye.txt contains the camera poses of each viewpoint (each line contains pose in TUM format: id, tx, ty, tz, rx, ry, rz, rw with id being the current view, tx,ty,tz the translation, rx, ry, rz, rw the rotation as quaternion).
• poses.yaml contains the scene objects annotation in the same world reference frame as the camera poses