What are the optimal algorithms for learning from data? Have we found them already, or are better ones out there to be discovered? Making these questions precise, and answering them, requires taking on the mathematically deep interplay between statistical and computational considerations. It also requires reconciling our theoretical toolbox with surprising new phenomena arising from practice, which seem to violate conventional rules of thumb regarding algorithm and model design. I will discuss progress along these lines: in terms of designing new algorithms for basic learning problems, controlling generalization in large statistical models, and understanding statistical questions arising from generative modeling.