In pursuit of a novel forecasting strategy for the energy market, we propose a ResNet-inspired model which estimates solar and wind energy production using weather images. The model is designed to capture high-frequency details while producing realistically smooth energy production profiles. To this end, we show the value of including multiple weather images from times preceding the estimation time, and demonstrate that the model outperforms traditional deep learning techniques and alternative state-of-the-art computer vision methods. Training and testing are performed on a novel data set that focuses on the state of California and spans the year 2019. The dataset, which is sourced from NOAA and CAISO, is a secondary contribution of this work. Finally, multiple topics in line with the motivation are proposed for future work.