The amount of energy generation from renewable energy sources, particularly from wind and photovoltaic plants, has seen a rapid rise in the last decade. Reliable and economic operation of power systems thus requires an accurate estimate of the power generated from renewable generation plants, particularly those that are intermittent in nature. This has accentuated the need to find an efficient and scalable scheme for forecasting meteorological parameters, such as solar radiation, with better accuracy. For short-term solar irradiance forecasting, the traditional point forecasting methods are rendered less useful due to the non-stationary characteristic of solar power. In this research work, we propose a unified architecture for multi-time-scale predictions for intra-day solar irradiance forecasting using recurrent neural networks (RNN) and long-short-term memory networks (LSTMs). This paper also lays out a framework for extending this modeling approach to intra-hour forecasting horizons, thus making it a multi-time-horizon forecasting approach capable of predicting intra-hour as well as intra-day solar irradiance. We develop an end-to-end pipeline to effectuate the proposed architecture. The robustness of the approach is demonstrated with case studies conducted for geographically scattered sites across the United States. The predictions demonstrate that our proposed unified architecture based approach is effective for multi-time-scale solar forecasts and achieves a lower root-mean-square prediction error when benchmarked against the best-performing methods documented in the literature that use separate models for each time-scale during the day. The proposed method enables multi-time-horizon forecasts with real-time inputs, which have a significant potential for practical industry applications in the evolving grid.