Esports has gained global popularity in recent years and several companies have started offering live streaming videos of esports games and events. This creates opportunities to develop large scale video understanding systems for new product features and services. We present a technique for detecting highlights from live streaming videos of esports game matches. Most video games use pronounced visual effects to emphasize highlight moments; we use CNNs to learn convolution filters of those visual effects for detecting highlights. We propose a cascaded prediction approach that allows us to deal with several challenges arise in a production environment. We demonstrate our technique on our new dataset of three popular game titles, Heroes of the Storm, League of Legends, and Dota 2. Our technique achieves 18 FPS on a single CPU with an average precision of up to 83.18%. Part of our technique is currently deployed in production on Yahoo Esports.