Non-IID data and Continual Learning processes in Federated Learning: A long road ahead

Federated Learning is a novel framework that allows multiple devices or institutions to train a machine learning model collaboratively while preserving their data private. This decentralized approach is prone to suffer the consequences of data statistical heterogeneity, both across the different entities and over time, which may lead to a lack of convergence. To avoid such issues, different methods have been proposed in the past few years. However, data may be heterogeneous in lots of different ways, and current proposals do not always determine the kind of heterogeneity they are considering. In this work, we formally classify data statistical heterogeneity and review the most remarkable learning strategies that are able to face it. At the same time, we introduce approaches from other machine learning frameworks, such as Continual Learning, that also deal with data heterogeneity and could be easily adapted to the Federated Learning settings.

Concept drift detection and adaptation for federated and continual learning

Smart devices, such as smartphones, wearables, robots, and others, can collect vast amounts of data from their environment. This data is suitable for training machine learning models, which can significantly improve their behavior, and therefore, the user experience. Federated learning is a young and popular framework that allows multiple distributed devices to train deep learning models collaboratively while preserving data privacy. Nevertheless, this approach may not be optimal for scenarios where data distribution is non-identical among the participants or changes over time, causing what is known as concept drift. Little research has yet been done in this field, but this kind of situation is quite frequent in real life and poses new challenges to both continual and federated learning. Therefore, in this work, we present a new method, called Concept-Drift-Aware Federated Averaging (CDA-FedAvg). Our proposal is an extension of the most popular federated algorithm, Federated Averaging (FedAvg), enhancing it for continual adaptation under concept drift. We empirically demonstrate the weaknesses of regular FedAvg and prove that CDA-FedAvg outperforms it in this type of scenario.

Incremental Learning from Low-labelled Stream Data in Open-Set Video Face Recognition

Deep Learning approaches have brought solutions, with impressive performance, to general classification problems where wealthy of annotated data are provided for training. In contrast, less progress has been made in continual learning of a set of non-stationary classes, mainly when applied to unsupervised problems with streaming data. Here, we propose a novel incremental learning approach which combines a deep features encoder with an Open-Set Dynamic Ensembles of SVM, to tackle the problem of identifying individuals of interest (IoI) from streaming face data. From a simple weak classifier trained on a few video-frames, our method can use unsupervised operational data to enhance recognition. Our approach adapts to new patterns avoiding catastrophic forgetting and partially heals itself from miss-adaptation. Besides, to better comply with real world conditions, the system was designed to operate in an open-set setting. Results show a benefit of up to 15% F1-score increase respect to non-adaptive state-of-the-art methods.

Collaborative and continual learning for classification tasks in a society of devices

Today we live in a context in which devices are increasingly interconnected and sensorized and are almost ubiquitous. Deep learning has become in recent years a popular way to extract knowledge from the huge amount of data that these devices are able to collect. Nevertheless, state-of-the-art learning methods have a number of drawbacks when facing real distributed problems, in which the available information is usually partial, biased and evolving over time. Moreover, if there is something that characterizes this society of devices is its high heterogeneity and dynamism, both in terms of users and the hardware itself. Therefore, against the tendency to centralize learning, in this work we want to present a new paradigm of learning in society, where devices get certain prominence back, having to learn in real time, locally, continuously, autonomously and from users, but also improving models globally, in the cloud, combining what is learned locally, in the devices. Hence, learning is carried out in a cyclical process of global consensus and local adaptation that can be repeated indefinitely over time. In this work we present a first architecture for this paradigm, which we call "glocal" learning. In order to test our proposal, we have applied it in a heterogeneous community of smartphone users to solve the problem of walking recognition. The results show the advantages that "glocal" learning provides with respect to other state-of-the-art methods.