TL;DR:A large scale, multi-modal dataset of high-fidelity dexterous grasping sequences featuring both human and robotic hands
Abstract
We present HRDexDB, a large scale, multi-modal dataset of high-fidelity dexterous grasping sequences
featuring both human and robotic hands. Unlike existing datasets, HRDexDB provides a comprehensive mapping
between human and robotic domains across 100 diverse objects. Our pipeline integrates state-of-the-art vision methods,
including HaMeR-based 3D human hand reconstruction and FoundationPose-based robust 6D object estimation, ensuring
high-precision spatial ground truth for both the agent and the manipulated object. To facilitate the study of physical
interaction, we capture paired robotic sequences via teleoperation, supplemented by high-resolution tactile feedback
and synchronized multi-view video. Furthermore, we address the common issue of kinematic model inaccuracies
by utilizing a calibrated multi-camera array to refine the robot's 3D pose in world coordinates.
HRDexDB comprises thousands of successful and failed grasping trials, each rich with visual, kinematic,
and tactile modalities. By providing dense correspondences between human dexterity and robotic perception,
HRDexDB serves as a primary benchmark for multi-modal policy learning, 6D object tracking, and cross-domain dexterous manipulation.
Dataset Overview
HRDexDB is the first large-scale dataset featuring paired human and dexterous robotic hand manipulation.
It provides over 1.4K sequences across 100 diverse objects and 4 distinct embodiments, all captured by a fully synchronized 23-camera system.
We provide detailed 3D annotations for every sequence. You can see a sample visualization of our dataset below.
Human dexterous grasping sequence from HRDexDB.
Robot dexterous grasping sequence from HRDexDB.
Unlike other existing robot datasets, HRDexDB provides reach tactile sensor signals for the Inspire robotic hand series.
Tactile sensing stream synchronized with the grasping sequence.
OMNIDEX: A Unified Multi-Modal Data Capture System
To construct HRDexDB, we developed OMNIDEX, a unified multi-modal capture platform. It features a dense rig of 23 synchronized cameras, integrated with real-time robot proprioception. This setup is specifically engineered to overcome severe occlusions during interaction with objects.
To construct HRDexDB, we developed OMNIDEX, a unified multi-modal capture platform. It features a dense rig of 23 synchronized cameras, integrated with real-time robot proprioception. This setup is specifically engineered to overcome severe occlusions during interaction with objects.