Overcoming Data Scarcity in IMU-based Human Motion Analysis

Abstract

Deep learning techniques enable the automatic analysis and interpretation of human motion from wearable sensor. However, despite extensive research efforts, the absence of vast cleanly labeled human motion sensor data in free-living setting still hindered the applications of deep models in the real world. Human motion data is known to have a large variance among subjects and devices. As the result, there is a significant performance gap between models trained with and without part of test subjects' data. In addition, due to the difficulty in labeling wearable sensor data post hoc, existing public datasets are either collected from scripted activities under controlled settings or contain severe label noises when collected through crowdsourcing. Moreover, since collecting motion data from frail populations such as older adults with impaired mobility can be physically demanding or even cause safety concerns, the data scarcity problem becomes more severe. In this dissertation, we aim to address these challenges through a multi-pronged approach.

First, we investigate domain adaptation techniques to handle the subject variance and device diversity in wearable sensor-based human activity recognition (HAR). We propose an invariant feature learning framework (IFLF) that extracts common information shared across subjects and devices. It incorporates two learning paradigms: 1) meta-learning to capture robust features across multiple source domains and adapt trained model to a target domain with similarity-based data selection; and 2) multi-task learning to deal with data shortage and enhance overall performance via knowledge sharing among different domains. Experimental results demonstrate that IFLF is effective in handling both subject and device variations across popular open datasets and an in-house dataset from older adults.

Inertial measurement units (IMU) datasets collected in naturalistic settings are often fraught with labeling noise due to misaligned onsets, the presence of concurrent activities, unpredictable terrains or human errors. However, state-of-the-art learning with label noise (LNL) approaches fail to converge due to the presence of subject variations. As a second contribution, we propose VALERIAN, an invariant feature learning for in-the-wild domain adaptation method for wearable sensor-based HAR. It consists of three components: self-supervised pre-training, invariant feature learning with noisy labels, and fast adaptation to new subjects. By training a multi-task model with separate task-specific layers for each subject, VALERIAN allows noisy labels to be dealt with individually for each subject while benefiting from shared feature representation across subjects. Experimental results show that VALERIAN significantly outperforms baseline approaches.

Simulating IMU data from other input modalities offers an alternative way to mitigate the wearable data scarcity problem. As a third contribution, we design CROMOSim, a cross-modality sensor simulator that synthesizes high fidelity virtual IMU data from data collected with motion capture systems or monocular RGB cameras. It utilizes a skinned multi-person linear model for 3D body pose and shape representations to enable simulating motions from arbitrary on-body positions. A deep learning model is used to learn the functional mapping from imperfect trajectory estimations in a 3D body tri-mesh representation to IMU data. Extensive empirical evidence demonstrates the high fidelity and utility of CROMOSim simulated data in downstream human motion analysis tasks include HAR and human pose estimation.