🤖 HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments

TL;DR: A real-world 4D dexterous grasping dataset capturing hand-object interactions across human and multiple robot embodiments

Abstract

We present HRDexDB, a real-world 4D dexterous grasping dataset capturing 3D hand-object interaction trajectories over time across five embodiments. The dataset comprises 3.2K trials over 100 diverse objects. Using a synchronized multi-camera system and an integrated reconstruction pipeline, HRDexDB provides multi-view and egocentric RGB observations, 3D hand geometry, robot states, and object 6D pose trajectories, together with success/failure annotations. Human and robotic hands interact with shared objects, enabling the study of embodiment-dependent grasp strategies and contact patterns. We demonstrate the dataset’s utility through human-to-robot contact map transfer, visual robot-object contact estimation, and retrieval-assisted grasping. Together, these results establish HRDexDB as a resource for studying and learning dexterous interactions across human and robotic embodiments.

Dataset Overview

HRDexDB is a real-world 4D dexterous grasping dataset capturing human and robotic hand-object interactions. It provides 3.2K trials across 100 diverse objects and 5 distinct embodiments (one human hand and four robotic hands), all captured by a fully synchronized 23-camera system. We provide detailed 3D annotations for every sequence. You can see a sample visualization of our dataset below.

3.2K

trials

100

objects

embodiments

Human Robot

High-fidelity human dexterous grasping sequences captured across diverse objects.

Paired robot hand executions recorded with synchronized visual and kinematic modalities.

3D Annotations

Paired human and robot annotations visualized across synchronized views.

Contact Visualization

Contact visualization highlighting interaction regions during grasping.

A Unified Multi-Modal Data Capture System

To construct HRDexDB, we developed a unified multi-modal capture platform. It features a dense rig of 23 synchronized cameras, integrated with real-time robot proprioception. This setup is specifically engineered to overcome severe occlusions during interaction with objects.

Contact

Send any comments or questions to Jongbin Lim: whdqls0534@snu.ac.kr or Taeyun Ha: taeyun012@snu.ac.kr.

Citation

@misc{lim2026hrdexdb,
      title={HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments},
      author={Jongbin Lim and Taeyun Ha and Seongho Cha and Kanghyeon Cho and Mingi Choi and Subin Jeon and Jisoo Kim and Byungjun Kim and Hanbyul Joo},
      year={2026},
      eprint={2604.14944},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2604.14944}, 
}