Abstract: In the classical M-estimation problem
we solve the (M) equation
\[
min_\beta \sum_{i=1}^n \rho(Y_i - X_i^T \beta) \] where each $X_i$ is a $p$-dimensional vector of predictors and $\beta in R^p$ is a $p$-dimensional vector of regression coefficients.
Under the classical Huber theory, we let $p$ be fixed and let $n \rightarrow \infty$. In recent years, it became popular to study the 'Big Data Asymptotic' where $n$ and $p_n$ are both large, and the design matrix $X$ is an iid Random design.

Under this asymptotic framework, Noureddine El Karoui, Peter Bickel, Bin Yu and co-authors have observed in a recent PNAS article that the classical Huber formula for the asymptotic variance is no longer correct, and new formulas were supplied based on persuasive heuristics and simulations.

In a recent ArXiv posting, Andrea Montanari and the presenter characterize the asymptotic variance formula using rigorous methods they and co-authors previously developed for the study of Lasso under high-dimensional asymptotics. The method involves a new algorithm for M-estimation called AMP (Approximate Message Passing) and a formal procedure to derive the asymptotic variance of the M-estimate, called State Evolution.

The core question to be understood here is what we call the {\it extra noise phenomenon}, that a new source of noise becomes non-negligible (and important!) under big data asymptotics that was negligible under classical asymptotics.
Our approach is heuristically very clear in showing the origin of this extra noise, showing its appreciable size, and allows many new insights, some of which we will review.

We will also show how these results for (M)-estimates are inextricably linked with our previous results on Lasso under high-dimensional asymptotics.

This talk covers joint work with Andrea Montanari, and draws on several papers, including some involving Montanari's coauthors Mohsen Bayati, Arian Maleki, and Adel Javanmard