3D HAND SHAPE AND MOTION RECONSTRUCTION FROM 2D VIDEOS
Loading...
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Department of Computing and Software, McMaster University
Abstract
Hand tracking and 3D reconstruction have emerged as crucial technologies in fields like computer vision, augmented reality, and human-computer interaction. This project aims to create a robust pipeline that transforms 2D hand video sequences into accurate 3D models of hands and their motions. The system leverages the Attention Collaboration-based Regressor (ACR) for two-hand reconstruction and the MANO model for hand shape and pose estimation. Through methods like 2D hand detection, keypoint estimation, and temporal optimization, the system enables smooth and realistic hand movement reconstruction. This technology has practical applications in areas such as sign language interpretation, virtual reality, and robotic control, overcoming challenges like hand occlusion, fast movements, and complex interactions. The results demonstrate significant accuracy and temporal consistency, making the system well-suited for real-time applications.
Description
This project focuses on developing a state-of-the-art system to convert 2D hand video inputs into detailed 3D hand shape and motion reconstructions. Using advanced deep learning techniques such as the Attention Collaboration-based Regressor (ACR) and the MANO hand model, the system achieves high accuracy in tracking and modeling hand movements. It is designed for applications in sign language interpretation, virtual reality, and human-computer interaction. Optimized for real-time or near-real-time performance, the system handles both hands simultaneously, delivering anatomically accurate and smooth hand motions, even in complex scenarios like occlusions and fast movements. As this is a cutting-edge system, further optimizations will be pursued to enhance accuracy, speed, and versatility for broader applications.