Resilient Control of Networked Microgrids using Vertical Federated Reinforcement Learning: Designs and Real-Time Test-Bed Validations

Improving system-level resiliency of networked microgrids is an important aspect with increased population of inverter-based resources (IBRs). This paper (1) presents resilient control design in presence of adversarial cyber-events, and proposes a novel federated reinforcement learning (Fed-RL) approach to tackle (a) model complexities, unknown dynamical behaviors of IBR devices, (b) privacy issues regarding data sharing in multi-party-owned networked grids, and (2) transfers learned controls from simulation to hardware-in-the-loop test-bed, thereby bridging the gap between simulation and real world. With these multi-prong objectives, first, we formulate a reinforcement learning (RL) training setup generating episodic trajectories with adversaries (attack signal) injected at the primary controllers of the grid forming (GFM) inverters where RL agents (or controllers) are being trained to mitigate the injected attacks. For networked microgrids, the horizontal Fed-RL method involving distinct independent environments is not appropriate, leading us to develop vertical variant Federated Soft Actor-Critic (FedSAC) algorithm to grasp the interconnected dynamics of networked microgrid. Next, utilizing OpenAI Gym interface, we built a custom simulation set-up in GridLAB-D/HELICS co-simulation platform, named Resilient RL Co-simulation (ResRLCoSIM), to train the RL agents with IEEE 123-bus benchmark test systems comprising 3 interconnected microgrids. Finally, the learned policies in simulation world are transferred to the real-time hardware-in-the-loop test-bed set-up developed using high-fidelity Hypersim platform. Experiments show that the simulator-trained RL controllers produce convincing results with the real-time test-bed set-up, validating the minimization of sim-to-real gap.

Enhancing Cyber Resilience of Networked Microgrids using Vertical Federated Reinforcement Learning

This paper presents a novel federated reinforcement learning (Fed-RL) methodology to enhance the cyber resiliency of networked microgrids. We formulate a resilient reinforcement learning (RL) training setup which (a) generates episodic trajectories injecting adversarial actions at primary control reference signals of the grid forming (GFM) inverters and (b) trains the RL agents (or controllers) to alleviate the impact of the injected adversaries. To circumvent data-sharing issues and concerns for proprietary privacy in multi-party-owned networked grids, we bring in the aspects of federated machine learning and propose a novel Fed-RL algorithm to train the RL agents. To this end, the conventional horizontal Fed-RL approaches using decoupled independent environments fail to capture the coupled dynamics in a networked microgrid, which leads us to propose a multi-agent vertically federated variation of actor-critic algorithms, namely federated soft actor-critic (FedSAC) algorithm. We created a customized simulation setup encapsulating microgrid dynamics in the GridLAB-D/HELICS co-simulation platform compatible with the OpenAI Gym interface for training RL agents. Finally, the proposed methodology is validated with numerical examples of modified IEEE 123-bus benchmark test systems consisting of three coupled microgrids.

Harmonic Modeling, Data Generation, and Analysis of Power Electronics-Interfaced Residential Loads

Integration of electronics-based residential appliances and distributed energy resources in homes is expected to rise with grid decarbonization. These devices may introduce significant harmonics into power networks that need to be closely studied in order to accurately model and forecast load. However, it can be difficult to obtain harmonic-rich voltage and current data -- necessary for identifying accurate load models -- for residential electrical loads. Recognizing this need, first a set of electronics-based end-use loads is identified and modeled in an electromagnetic transients program tool for a residence. Second, an impedance-varying method is proposed to generate harmonic data that captures harmonic propagation to the supply voltage and harmonic interactions among end-use loads connected to the same supply voltage. Third, a harmonic-rich dataset produced via the proposed methodology is demonstrated to successfully identify frequency coupling matrix-based harmonic load models using the least-squares method. Numerical results demonstrate the accuracy of the model. The impact of limited data availability on model identification is also explored.

Application and Comparison of Deep Learning Methods in the Prediction of RNA Sequence Degradation and Stability

mRNA vaccines are receiving increased interest as potential alternatives to conventional methods for the prevention of several diseases, including Covid-19. This paper proposes and evaluates three deep learning models (Long Short Term Memory networks, Gated Recurrent Unit networks, and Graph Convolutional Networks) as a method to predict the stability/reactivity and risk of degradation of sequences of RNA. Reasonably accurate results were able to be generated, with the Graph Convolutional Network being the best predictor of reactivity (RMSE = 0.249) while the Gated Recurrent Unit Network was the best at predicting risks of degradation under various circumstances (RMSE = 0.266). Results suggest feasibility of applying such methods in mRNA vaccine research in the near future.