Hierarchical Approaches for Generating Stylistic Human Motions from Audio

Abstract

Generating realistic and plausible human motions driven by audio has been explored for decades and has a wide range of applications in real life, such as virtual reality, gaming, films, and human-computer interaction. However, the spatiotemporal complexity of human motion and the requirement for generated movements to align with audio features pose significant challenges to motion synthesis algorithms. In this thesis, we first explore the approaches to human motion learning and synthesis, with a particular focus on data-driven dance and gesture generation. Then we propose a two-level framework to synthesize dance and gesture motions based on the music and speech input respectively, where the high-level motion planner models the overall structure of the motion sequence and the low-level implementer incorporates audio influences to generate detailed movements with nuances. Moreover, we introduce a motion editing module that refines generated dance. Building on this, we propose a method for mapping user inputs to dance style, giving users further control over the generated dance, as well as a data-driven approach for mapping speech audio to gesture style. Extensive experiments conducted on two datasets demonstrate the superior effectiveness and efficiency of our proposed method in generating diverse, high-quality dance and gesture motions that are well-synchronized with audio inputs.