Distributional Reinforcement Learning (RL) learns the whole conditional distribution of costs-to-go, given state and action, but then only ever looks at the mean (e.g., C51, IQN). While this appears unnecessary on its face, empirically it often improves on analogous approaches (e.g., DQN) that directly learn only the conditional mean (i.e., the Q-function). A principled understanding as to why and when this happens has been elusive. In this talk, I resolve this question by showing that distributional RL enjoys first- and second-order regret bounds in both online and offline RL in general MDPs with function approximation. In many cases these are the first bounds of their kind for any RL algorithm, distributional or otherwise. First-order bounds scale with optimal average costs and, e.g., establish fast convergence in goal-based tasks when the optimal policy reliably reaches the goal. Second-order bounds scale with the variance of cumulative costs and, e.g., establish fast convergence under low stochasticity, as is often encountered in robotics. I explain how this phenomenon arises due to automatic adaptation to heteroskedasticity, in contrast to previous heuristic explanations for the success of distributional RL. Going beyond risk-neutral RL, where the benefits of being distributional may be surprising, I will conclude with new results for sharp policy evaluation bounds in the presence of unobserved confounding and for minimax optimal risk-sensitive RL, two settings where being distributional is clearly a necessity.

Bio:  Nathan Kallus is an Associate Professor in the School of Operations Research and Information Engineering and Cornell Tech at Cornell University. Nathan’s research interests include personalization; optimization, especially under uncertainty; causal inference; sequential decision making; credible and robust inference; and algorithmic fairness. He holds a PhD in Operations Research from MIT as well as a BA in Mathematics and a BS in Computer Science both from UC Berkeley. Before coming to Cornell, Nathan was a Visiting Scholar at USC’s Department of Data Sciences and Operations and a Postdoctoral Associate at MIT’s Operations Research and Statistics group.