Worm-level Control through Search-based Reinforcement Learning

Through natural evolution, nervous systems of organisms formed near-optimal structures to express behavior. Here, we propose an effective way to create control agents, by \textit{re-purposing} the function of biological neural circuit models, to govern similar real world applications. We model the tap-withdrawal (TW) neural circuit of the nematode, \textit{C. elegans}, a circuit responsible for the worm's reflexive response to external mechanical touch stimulations, and learn its synaptic and neural parameters as a policy for controlling the inverted pendulum problem. For reconfiguration of the purpose of the TW neural circuit, we manipulate a search-based reinforcement learning. We show that our neural policy performs as good as existing traditional control theory and machine learning approaches. A video demonstration of the performance of our method can be accessed at \url{https://youtu.be/o-Ia5IVyff8}.

Searching for Biophysically Realistic Parameters for Dynamic Neuron Models by Genetic Algorithms from Calcium Imaging Recording

Individual Neurons in the nervous systems exploit various dynamics. To capture these dynamics for single neurons, we tune the parameters of an electrophysiological model of nerve cells, to fit experimental data obtained by calcium imaging. A search for the biophysical parameters of this model is performed by means of a genetic algorithm, where the model neuron is exposed to a predefined input current representing overall inputs from other parts of the nervous system. The algorithm is then constrained for keeping the ion-channel currents within reasonable ranges, while producing the best fit to a calcium imaging time series of the AVA interneuron, from the brain of the soil-worm, C. elegans. Our settings enable us to project a set of biophysical parameters to the the neuron kinetics observed in neuronal imaging.

Non-Associative Learning Representation in the Nervous System of the Nematode Caenorhabditis elegans

Caenorhabditis elegans (C. elegans) illustrated remarkable behavioral plasticities including complex non-associative and associative learning representations. Understanding the principles of such mechanisms presumably leads to constructive inspirations for the design of efficient learning algorithms. In the present study, we postulate a novel approach on modeling single neurons and synapses to study the mechanisms underlying learning in the C. elegans nervous system. In this regard, we construct a precise mathematical model of sensory neurons where we include multi-scale details from genes, ion channels and ion pumps, together with a dynamic model of synapses comprised of neurotransmitters and receptors kinetics. We recapitulate mechanosensory habituation mechanism, a non-associative learning process, in which elements of the neural network tune their parameters as a result of repeated input stimuli. Accordingly, we quantitatively demonstrate the roots of such plasticity in the neuronal and synaptic-level representations. Our findings can potentially give rise to the development of new bio-inspired learning algorithms.

SIM-CE: An Advanced Simulink Platform for Studying the Brain of Caenorhabditis elegans

We introduce SIM-CE, an advanced, user-friendly modeling and simulation environment in Simulink for performing multi-scale behavioral analysis of the nervous system of Caenorhabditis elegans (C. elegans). SIM-CE contains an implementation of the mathematical models of C. elegans's neurons and synapses, in Simulink, which can be easily extended and particularized by the user. The Simulink model is able to capture both complex dynamics of ion channels and additional biophysical detail such as intracellular calcium concentration. We demonstrate the performance of SIM-CE by carrying out neuronal, synaptic and neural-circuit-level behavioral simulations. Such environment enables the user to capture unknown properties of the neural circuits, test hypotheses and determine the origin of many behavioral plasticities exhibited by the worm.

An Automated Auto-encoder Correlation-based Health-Monitoring and Prognostic Method for Machine Bearings

This paper studies an intelligent ultimate technique for health-monitoring and prognostic of common rotary machine components, particularly bearings. During a run-to-failure experiment, rich unsupervised features from vibration sensory data are extracted by a trained sparse auto-encoder. Then, the correlation of the extracted attributes of the initial samples (presumably healthy at the beginning of the test) with the succeeding samples is calculated and passed through a moving-average filter. The normalized output is named auto-encoder correlation-based (AEC) rate which stands for an informative attribute of the system depicting its health status and precisely identifying the degradation starting point. We show that AEC technique well-generalizes in several run-to-failure tests. AEC collects rich unsupervised features form the vibration data fully autonomous. We demonstrate the superiority of the AEC over many other state-of-the-art approaches for the health monitoring and prognostic of machine bearings.

Extracting Implicit Social Relation for Social Recommendation Techniques in User Rating Prediction

Recommendation plays an increasingly important role in our daily lives. Recommender systems automatically suggest items to users that might be interesting for them. Recent studies illustrate that incorporating social trust in Matrix Factorization methods demonstrably improves accuracy of rating prediction. Such approaches mainly use the trust scores explicitly expressed by users. However, it is often challenging to have users provide explicit trust scores of each other. There exist quite a few works, which propose Trust Metrics to compute and predict trust scores between users based on their interactions. In this paper, first we present how social relation can be extracted from users' ratings to items by describing Hellinger distance between users in recommender systems. Then, we propose to incorporate the predicted trust scores into social matrix factorization models. By analyzing social relation extraction from three well-known real-world datasets, which both: trust and recommendation data available, we conclude that using the implicit social relation in social recommendation techniques has almost the same performance compared to the actual trust scores explicitly expressed by users. Hence, we build our method, called Hell-TrustSVD, on top of the state-of-the-art social recommendation technique to incorporate both the extracted implicit social relations and ratings given by users on the prediction of items for an active user. To the best of our knowledge, this is the first work to extend TrustSVD with extracted social trust information. The experimental results support the idea of employing implicit trust into matrix factorization whenever explicit trust is not available, can perform much better than the state-of-the-art approaches in user rating prediction.

ARES: Adaptive Receding-Horizon Synthesis of Optimal Plans

We introduce ARES, an efficient approximation algorithm for generating optimal plans (action sequences) that take an initial state of a Markov Decision Process (MDP) to a state whose cost is below a specified (convergence) threshold. ARES uses Particle Swarm Optimization, with adaptive sizing for both the receding horizon and the particle swarm. Inspired by Importance Splitting, the length of the horizon and the number of particles are chosen such that at least one particle reaches a next-level state, that is, a state where the cost decreases by a required delta from the previous-level state. The level relation on states and the plans constructed by ARES implicitly define a Lyapunov function and an optimal policy, respectively, both of which could be explicitly generated by applying ARES to all states of the MDP, up to some topological equivalence relation. We also assess the effectiveness of ARES by statistically evaluating its rate of success in generating optimal plans. The ARES algorithm resulted from our desire to clarify if flying in V-formation is a flocking policy that optimizes energy conservation, clear view, and velocity alignment. That is, we were interested to see if one could find optimal plans that bring a flock from an arbitrary initial state to a state exhibiting a single connected V-formation. For flocks with 7 birds, ARES is able to generate a plan that leads to a V-formation in 95% of the 8,000 random initial configurations within 63 seconds, on average. ARES can also be easily customized into a model-predictive controller (MPC) with an adaptive receding horizon and statistical guarantees of convergence. To the best of our knowledge, our adaptive-sizing approach is the first to provide convergence guarantees in receding-horizon techniques.

Deep Neural Programs for Adaptive Control in Cyber-Physical Systems

We introduce Deep Neural Programs (DNP), a novel programming paradigm for writing adaptive controllers for cy-ber-physical systems (CPS). DNP replace if and while statements, whose discontinuity is responsible for undecidability in CPS analysis, intractability in CPS design, and frailness in CPS implementation, with their smooth, neural nif and nwhile counterparts. This not only makes CPS analysis decidable and CPS design tractable, but also allows to write robust and adaptive CPS code. In DNP the connection between the sigmoidal guards of the nif and nwhile statements has to be given as a Gaussian Bayesian network, which reflects the partial knowledge, the CPS program has about its environment. To the best of our knowledge, DNP are the first approach linking neural networks to programs, in a way that makes explicit the meaning of the network. In order to prove and validate the usefulness of DNP, we use them to write and learn an adaptive CPS controller for the parallel parking of the Pioneer rovers available in our CPS lab.

Monitoring with uncertainty

We discuss the problem of runtime verification of an instrumented program that misses to emit and to monitor some events. These gaps can occur when a monitoring overhead control mechanism is introduced to disable the monitor of an application with real-time constraints. We show how to use statistical models to learn the application behavior and to "fill in" the introduced gaps. Finally, we present and discuss some techniques developed in the last three years to estimate the probability that a property of interest is violated in the presence of an incomplete trace.