Advanced Topics in Estimation and Information Theory

Abstract

<p>The main theme of this dissertation is statistical estimation and information theory. There are three related topics including "distributed estimation", "an information geometric approach to ML estimation with incomplete data" and "joint identification and estimation in non-linear state space using Bayesian filters". The expectationmaximization (EM) algorithm, as an iterative estimation technique for dealing with incomplete data is the common bond that binds these three topics together.</p> <p>1. <em>Distributed estimation</em></p> <p>Distributed estimation involves the study of estimation theory in an information theoretic framework. This field concerns the following question: "What if the purpose of communications in a distributed environment is parameter estimation rather than source reconstruction?" The first part of this thesis is dedicated to designing low-complexity iterative algorithms for distributed estimation. The algorithm design, in this case, involves transmission of statistics via communication systems. Therefore, the first question raised is "whether the code rates in distributed estimation are different from those in conventional communications?" Surprisingly, under certain conditions, the answer is found to be negative. It is shown that for fixed parameters, the achievable rates coincide with rates in conventional distributed coding of correlated sources (i.e. Slepian-Wolf region). In order to prove the main theorem, we also devise a novel distributed binning scheme and a new theorem in Large deviation theory that are used for proving our distributed coding theorem. The proof of the converse is implemented by a generalized <em>Fano's inequality</em> for distributed estimation.</p> <p>Determination of the region of achievable rates for efficient estimation of a general source is an extremely difficult problem. This fact is the motivation for proving a theorem that provides a method for determining the region of achievable rates for a large class of sources with a convex mutual information with respect to the unknown parameters.</p> <p>With a given set of rates, an efficient implementation of universal coding schemes for distributed estimation based on the expectation maximization (EM) technique is presented. Since the correlation channel between the sources is assumed to be unknown at the joint decoder, previously proposed distributed coding schemes are not useful for this purpose. Therefore, LDPC-based coset-coding schemes are extended to the case where the correlation channel is unknown at the decoder. The basic idea is to implement a low-complexity version of the EM algorithm on a factor~graph that includes an LDPC decoding mechanism.</p> <p>2. <em>Information geometric approach to ML estimation with incomplete data</em></p> <p>The stochastic maximum likelihood estimation of parameters with incomplete data is cast in an information geometric framework. In this vein we develop the information geometric identification (IGID) algorithm, that provides an alternative iterative solution to the incomplete-data estimation problem. The algorithm consists of iterative alternating projections on two sets of probability distributions (PD); i.e., likelihood PD's and data empirical distributions. A Gaussian assumption on the source distribution permits a closed form lowcomplexity solution for these projections. The method is applicable to a wide range of problems; however the emphasis is on semi-blind identification of unknown parameters in a multi-input multi-output (MIMO) communications system.</p> <p>3. <em>Joint identification and estimation in non-linear state space using Bayesian filters</em></p> <p>There are situations in estimation where nonlinear state-space models where the model parameters or the model structure itself are not known a priori or are known only partially. In these scenarios, standard estimation algorithms like the extended Kalman Filter (EKF), which assume perfect knowledge of the model parameters, are not accurate. The nonlinear state estimation problem with possibly non-Gaussian noise in the presence of measurement model uncertainty is modeled as a special case of maximum likelihood estimation with incomplete data. The EM algorithm is used to solve the problem. The expectation (E) step is implemented by a particle filter that is initialized by a Monte-Carlo Markov chain algorithm. In the maximization (M) step, a nonlinear regression method, here using a mixture of Gaussians (MoG), is used to approximate (identify) the uncertain model equations. The proposed procedure is used to solve a highly nonlinear bearing-only tracking problem, as well as the sensor registration problem in a multi-sensor fusion scenario.</p>