Incentive mechanisms are central to the functionality of permissionless blockchains: they incentivize participants to run and secure the underlying consensus protocol. Designing incentive-compatible incentive mechanisms is notoriously challenging, however. Even systems with strong theoretical security guarantees in traditional settings, where users are either Byzantine or honest, often exclude analysis of rational users, who may exploit incentives to deviate from honest behavior. As a result, most public blockchains today use incentive mechanisms whose security properties are poorly understood and largely untested. In this work, we propose SquirRL, a framework for using deep reinforcement learning to identify attack strategies on blockchain incentive mechanisms. With minimal setup, SquirRL replicates known theoretical results on the Bitcoin protocol. In more complex and realistic settings, as when mining power varies over time, it identifies attack strategies superior to those known in the literature. Finally, SquirRL yields results suggesting that classical selfish mining attacks against Bitcoin lose effectiveness in the presence of multiple attackers. These results shed light on why selfish mining, which is unobserved to date in the wild, may be a poor attack strategy.