We are interested in the problem of unit-level counterfactual inference in the presence of unobserved confounders owing to the increasing importance of personalized decision-making in many domains: consider a recommender system interacting with a user over time where each user is provided recommendations based on observed demographics, prior engagement levels as well as certain unobserved factors. We model the underlying joint distribution through an exponential family. This reduces the task of unit-level counterfactual inference to simultaneously learning a collection of distributions of a given exponential family with different unknown parameters with single observation per distribution. We discuss a computationally efficient method for learning all of these parameters with estimation error scaling linearly with the metric entropy of the space of unknown parameters – if the parameters are s-sparse linear combination of k known vectors in p dimension, the error scales as O(s log k/p).  En route, we derive sufficient conditions for compactly supported distributions to satisfy the logarithmic Sobolev inequality.

Based on a joint work with Raaz Dwivedi (Cornell), Abhin Shah (MIT) and Greg Wornell (MIT).

Main paper: https://arxiv.org/abs/2211.08209

Related paper: https://arxiv.org/pdf/2309.06413

Bio: Devavrat Shah is Andrew (1956) and Erna Viterbi Professor of Electrical Engineering and Computer Science at MIT where he has been teaching since 2005. He was the faculty director of Deshpande Center for Tech Innovation and the founding director of the Statistics and Data Science Center at MIT. His current research interests include algorithms for causal inference, social data processing and stochastic networks. He is a distinguished alumni of his alma mater IIT Bombay. His work has been recognized through career prizes 2008 ACM Sigmetrics Rising Star and 2010 INFORMS Erlang Prize; paper prizes at IEEE Infocom, ACM Sigmetrics, NeurIPS, INFORMS Applied Probability Society, INFORM Management Science and Operations Management; INFORMS George B Dantzig thesis prize and test of time awards at ACM Sigmetrics. In 2013, he co-founded the machine learning start-up Celect (part of Nike) which helps retailers optimize inventory using accurate demand forecasting. In 2019, he co-founded Ikigai Labs with the mission of bringing AI to Enterprises.