Bayesian methods have great promise in big data sets, but this promise has not been fully realized due to the lack of scalable computational methods. Usual sampling algorithms bog down as the size of the data increase. For massive data sets, it has become routine to rely on penalized optimization approaches implemented on distributed computing systems. However, in scientific applications, it is crucial to obtain a good characterization of uncertainty and not simply a point estimate. To scale up Bayes inference to huge sample sizes, we propose to run MCMC (or an alternative sampling algorithm) in parallel for subsets of the data and then combine. The key question is how to best combine balancing computational efficiency and accuracy? We address this using a new approach based on estimating the Barycenter of subset posteriors under a Wasserstein metric. This can be done extremely quickly, and can produce an accurate approximation of the overall posterior or a robust alternative “median posterior” that accommodates data contamination and model misspecification. Some initial theory is presented, along with results in some interesting applications.
Joint with Sanvesh Srivastava