Deep Reinforcement Learning Based Networked Control with Network Delays for Signal Temporal Logic Specifications

We present a novel deep reinforcement learning (DRL)-based design of a networked controller with network delays for signal temporal logic (STL) specifications. We consider the case in which both the system dynamics and network delays are unknown. Because the satisfaction of an STL formula is based not only on the current state but also on the behavior of the system, we propose an extension of the Markov decision process (MDP), which is called a $\tau\delta$-MDP, such that we can evaluate the satisfaction of the STL formula under the network delays using the $\tau\delta$-MDP. Thereafter, we construct deep neural networks based on the $\tau\delta$-MDP and propose a learning algorithm. Through simulations, we also demonstrate the learning performance of the proposed algorithm.