In many real-world applications of reinforcement learning (RL) such as healthcare, dialogue systems and robotics, running a new policy on humans or robots can be costly or risky. This gives rise to the critical need for off-policy estimation, that is, estimate the average reward of a target policy given data that was previously collected by another policy. This talk will describe some recent advances for long- or even infinite-horizon off-policy estimation, where standard methods suffer a variance that grows exponentially with the horizon (“curse of horizon”).  The key to these methods is a duality structure in RL, whose use goes beyond off-policy estimation.