Model Predictive Control-Guided Reinforcement Learning for Implicit Balancing

In Europe, profit-seeking balance responsible parties can deviate in real time from their day-ahead nominations to assist transmission system operators in maintaining the supply-demand balance. Model predictive control (MPC) strategies to exploit these implicit balancing strategies capture arbitrage opportunities, but fail to accurately capture the price-formation process in the European imbalance markets and face high computational costs. Model-free reinforcement learning (RL) methods are fast to execute, but require data-intensive training and usually rely on real-time and historical data for decision-making. This paper proposes an MPC-guided RL method that combines the complementary strengths of both MPC and RL. The proposed method can effectively incorporate forecasts into the decision-making process (as in MPC), while maintaining the fast inference capability of RL. The performance of the proposed method is evaluated on the implicit balancing battery control problem using Belgian balancing data from 2023. First, we analyze the performance of the standalone state-of-the-art RL and MPC methods from various angles, to highlight their individual strengths and limitations. Next, we show an arbitrage profit benefit of the proposed MPC-guided RL method of 16.15% and 54.36%, compared to standalone RL and MPC.

Machine Learning for Improved Gas Network Models in Coordinated Energy Systems

The current energy transition promotes the convergence of operation between the power and natural gas systems. In that direction, it becomes paramount to improve the modeling of non-convex natural gas flow dynamics within the coordinated power and gas dispatch. In this work, we propose a neural-network-constrained optimization method which includes a regression model of the Weymouth equation, based on supervised machine learning. The Weymouth equation links gas flow to inlet and outlet pressures for each pipeline via a quadratic equality, which is captured by a neural network. The latter is encoded via a tractable mixed-integer linear program into the set of constraints. In addition, our proposed framework is capable of considering bidirectionality without having recourse to complex and potentially inaccurate convexification approaches. We further enhance our model by introducing a reformulation of the activation function, which improves the computational efficiency. An extensive numerical study based on the real-life Belgian power and gas systems shows that the proposed methodology yields promising results in terms of accuracy and tractability.