Representation learning is a promising approach for solving large-scale Reinforcement Learning (RL) problems where data is complex and high-dimensional (i.e., rich-observation RL). While representation learning in computer vision and natural language processing has made significant empirical progress, the understanding of its application to RL is still limited, due to the unique challenges from RL such as the interplay between representation learning and exploration (e.g., deep exploration requires informative representation which in turn cannot be easily discovered without adequate exploration). In this talk, we study representation learning for RL in the context of low-rank MDPs where features are unknown a priori, which forces us to think about the intricate coupling of representation learning and decision making. We develop two styles of representation learning methods, one is based on model learning, and the other one is based adversarial training. Both methods interleave representation learning, exploration, and exploitation to achieve polynomial sample complexity in learning near optimal policies. Empirically, we demonstrate our approach on a set of challenging rich observation RL benchmarks which require deep exploration, and show that our approach outperforms prior state-of-art theoretical approaches and also empirical deep RL baselines.

This is joint work with Masa Uehara, Xuezhou Zhang, Yuda Song, Mengdi Wang, and Alekh Agarwal.