Sample-based and Feature-based Federated Learning via Mini-batch SSCA

Due to the resource consumption for transmitting massive data and the concern for exposing sensitive data, it is impossible or undesirable to upload clients' local databases to a central server. Thus, federated learning has become a hot research area in enabling the collaborative training of machine learning models among multiple clients that hold sensitive local data. Nevertheless, unconstrained federated optimization has been studied mainly using stochastic gradient descent (SGD), which may converge slowly, and constrained federated optimization, which is more challenging, has not been investigated so far. This paper investigates sample-based and feature-based federated optimization, respectively, and considers both the unconstrained problem and the constrained problem for each of them. We propose federated learning algorithms using stochastic successive convex approximation (SSCA) and mini-batch techniques. We show that the proposed algorithms can preserve data privacy through the model aggregation mechanism, and their security can be enhanced via additional privacy mechanisms. We also show that the proposed algorithms converge to Karush-Kuhn-Tucker (KKT) points of the respective federated optimization problems. Besides, we customize the proposed algorithms to application examples and show that all updates have closed-form expressions. Finally, numerical experiments demonstrate the inherent advantages of the proposed algorithms in convergence speeds, communication costs, and model specifications.

Sample-based Federated Learning via Mini-batch SSCA

In this paper, we investigate unconstrained and constrained sample-based federated optimization, respectively. For each problem, we propose a privacy preserving algorithm using stochastic successive convex approximation (SSCA) techniques, and show that it can converge to a Karush-Kuhn-Tucker (KKT) point. To the best of our knowledge, SSCA has not been used for solving federated optimization, and federated optimization with nonconvex constraints has not been investigated. Next, we customize the two proposed SSCA-based algorithms to two application examples, and provide closed-form solutions for the respective approximate convex problems at each iteration of SSCA. Finally, numerical experiments demonstrate inherent advantages of the proposed algorithms in terms of convergence speed, communication cost and model specification.