Considerations about temporal rescaling, discretization, and linearization of RNNs

We explore the mathematical foundations of Recurrent Neural Networks (RNNs) and three fundamental procedures: temporal rescaling, discretization, and linearization. These techniques provide essential tools for characterizing RNN behaviour, enabling insights into temporal dynamics, practical computational implementation, and linear approximations for analysis. We discuss the flexible order of application of these procedures, emphasizing their significance in modelling and analyzing RNNs for computational neuroscience and machine learning applications. We explicitly describe here under what conditions these procedures can be interchangeable.

Recurrent Neural Networks as Electrical Networks, a formalization

Since the 1980s, and particularly with the Hopfield model, recurrent neural networks or RNN became a topic of great interest. The first works of neural networks consisted of simple systems of a few neurons that were commonly simulated through analogue electronic circuits. The passage from the equations to the circuits was carried out directly without justification and subsequent formalisation. The present work shows a way to formally obtain the equivalence between an analogue circuit and a neural network and formalizes the connection between both systems. We also show which are the properties that these electrical networks must satisfy. We can have confidence that the representation in terms of circuits is mathematically equivalent to the equations that represent the network.

What you need to know to train recurrent neural networks to make Flip Flops memories and more

Training neural networks to perform different tasks is relevant across various disciplines that go beyond Machine Learning. In particular, Recurrent Neural Networks (RNN) are of great interest to different scientific communities. Open-source frameworks dedicated to Machine Learning such as Tensorflow \cite{chollet2015keras} and Keras \cite{tensorflow2015-whitepaper} has produced significative changes in the development of technologies that we currently use. One relevant problem that can be approach is how to build the models for the study of dynamical systems, and how to extract the relevant information to be able to answer the scientific questions of interest. The purpose of the present work is to contribute to this aim by using a temporal processing task, in this case, a 3-bit Flip Flop memory, to show the modeling procedure in every step: from equations to the software code, using Tensorflow and Keras. The obtained networks are analyzed to describe the dynamics and to show different visualization and analysis tools. The code developed in this premier is provided to be used as a base for model other systems.