We propose an end-to-end approach for solving inverse problems for a class of complex astronomical signals, namely Spectral Energy Distributions (SEDs). Our goal is to reconstruct such signals from scarce and/or unreliable measurements. We achieve that by leveraging a learned structural prior in the form of a Deep Generative Network. Similar methods have been tested almost exclusively for images which display useful properties (e.g., locality, periodicity) that are implicitly exploited. However, SEDs lack such properties which make the problem more challenging. We manage to successfully extend the methods to SEDs using a Generative Latent Optimization model trained with significantly fewer and corrupted data.