Online EM for Exponential Family Distributions

Abstract

The expectation-maximization (EM) algorithm is a widely utilized method for fitting probabilistic models with missing data or latent variables. However, applying the EM algorithm to high-dimensional datasets requires storing the entire dataset in computer memory, which is necessary at each iteration. Fortunately, online learning algorithms are available that overcome this issue by processing data sequentially and updating model parameters at each step. In this thesis, we introduce an approach for online training of mixture models of exponential family distributions through modifications to the EM algorithm.

The first part of our contribution is the learning of Gaussian mixture models (GMMs) using alternating mirror descent updates. This method employs custom Bregman divergences for parameter updates and allows for the processing of one unit of data at each iteration. Our second contribution is an improved version of the EM algorithm for exponential family distributions. In this case, we replace the M-step updates with mirror descent updates and further enhance them by introducing a new Bregman divergence for the updates. Finally, we apply the improved EM algorithm to online principal component analysis (PCA) for exponential families, where the number of principal components is adaptively learned by pruning factors with low correlations to the observations. We evaluate the performance of our proposed approaches by comparing them with both simulation and real datasets.