HOnnotate: A method for 3D Annotation of Hand and Object Poses

Shreyas Hampali, Mahdi Rad, Markus Oberweger, Vincent Lepetit

In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, Seattle, WA, USA.


Abstract: We propose a method for annotating images of a hand manipulating an object with the 3D poses of both the hand and the object, together with a dataset created using this method. There is a current lack of annotated real images for this problem, as estimating the 3D poses is challenging, mostly because of the mutual occlusions between the hand and the object. To tackle this challenge, we capture sequences with one or several RGB-D cameras, and jointly optimizes the 3D hand and object poses over all the frames simultaneously. This method allows us to automatically annotate each frame with accurate estimates of the poses, despite large mutual occlusions. With this method, we created HO-3D, the first markerless dataset of color images with 3D annotations of both hand and object. This dataset is currently made of 80,000 frames, 65 sequences, 10 persons, and 10 objects. We also use it to train a deepnet to perform RGB-based single frame hand pose estimation and provide a baseline on our dataset.

[paper] [ Supplementary] [ Code]

Dataset: Please visit the project page to download the dataset

CodaLab Competition: Online challenge for hand pose estimation from single RGB image on our dataset here



  1. Shreyas Hampali, “Video Coding Including a Stage-Interdependent Multi-Stage Butterfly Integer Transform”, U.S. Patent 20160021369, published Jan 21, 2016.
  2. Shreyas Hampali, Ajit Rao, Yogesh Gupta and Conrad Harrison, “Artifact detection in a contrast enhanced output image”, U.S. Patent filed, Application no. 15/702,394
  3. Shreyas Hampali, Pawan Baheti and Naveen Srinivasamurthy, “Systems and methods for non-recursive image signal processor tuning using a reference image”, India Patent filed, Application no. 201841003400
  4. Shilpi Sahi, Pawan Baheti, Aarrushi Shandilya, Shreyas Hampali, Naveen Srinivasamurthy and Yogesh Gupta, “Systems and methods for assisted image signal processor tuning”, India Patent filed, Application no. 201841003395
  5. Pawan Baheti, Shilpi Sahu, Naveen Srinivasamurthy, Yogesh Gupta, Uday Kiran Pudipeddi, Shreyas Hampali, “Systems and methods for assisted image signal processor tuning using a reference image”, India patent filed, Application no. 201841003373
  6. Shreyas Hampali and Dowray Raghvendra Rao, ”Remote Image based Measurement System”, India Patent 4785/CHE/2012, filed November 2012.