Deep learning builds upon the mysterious abilities of gradient-based optimization algorithms. Not only can these algorithms often achieve low loss on complicated non-convex training objectives, but the solutions found can also generalize remarkably well to unseen test data despite significant over-parameterization of the models. Classical approaches in optimization and learning theory that treat empirical risk minimization as a black box are insufficient to explain these mysteries. In this talk, I will illustrate how we can make progress towards understanding deep learning by a more refined approach that opens the black box of the optimizer. In particular, I will present some recent results that take into account the trajectories taken by the gradient descent algorithm, including two case studies: (i) solving low-rank matrix completion via deep linear neural networks, (ii) the connection between wide neural networks and neural tangent kernels, and their implications.