Multi-modal Cross-masked Autoencoder for Digital Health Measurements

Abstract: The growing prevalence of digital health technologies has led to the generation of complex multi-modal data, such as physical activity measurements simultaneously collected from various sensors of mobile and wearable devices. These data hold immense potential for advancing health studies, but current methods predominantly rely on supervised learning, requiring extensive labeled datasets that are often expensive or impractical to obtain, especially in clinical studies. To address this limitation, we propose a self-supervised learning framework called Multi-modal Cross-masked Autoencoder (MoCA) that leverages cross-modality masking and the transformer autoencoder architecture to utilize both temporal correlations within modalities and cross-modal correlations between data streams. We also provide theoretical guarantees to support the effectiveness of the cross-modality masking scheme in MoCA. Comprehensive experiments and ablation studies demonstrate that our method outperforms existing approaches in both reconstruction and downstream tasks.

Bio: Jingjing Zou is an Assistant Professor of biostatistics at the Herbert Wertheim School of Public Health and Human Longevity Science, University of California at San Diego, La Jolla, CA, USA. She serves as the Co-Leader of the data management and statistics core at the UCSD Shiley-Marcos Alzheimer’s Disease Research Center and is affiliated with the Moores UCSD Cancer Center. Her research focuses on innovations in machine learning and functional data analysis, with a particular emphasis on their applications to complex challenges in health sciences.