Alert button

Categorical Foundations of Gradient-Based Learning

Mar 02, 2021
G. S. H. Cruttwell, Bruno Gavranović, Neil Ghani, Paul Wilson, Fabio Zanasi

Figure 1 for Categorical Foundations of Gradient-Based Learning
Figure 2 for Categorical Foundations of Gradient-Based Learning
Figure 3 for Categorical Foundations of Gradient-Based Learning
Figure 4 for Categorical Foundations of Gradient-Based Learning

Share this with someone who'll enjoy it:

We propose a categorical foundation of gradient-based machine learning algorithms in terms of lenses, parametrised maps, and reverse derivative categories. This foundation provides a powerful explanatory and unifying framework: it encompasses a variety of gradient descent algorithms such as ADAM, AdaGrad, and Nesterov momentum, as well as a variety of loss functions such as as MSE and Softmax cross-entropy, shedding new light on their similarities and differences. Our approach also generalises beyond neural networks (modelled in categories of smooth maps), accounting for other structures relevant to gradient-based learning such as boolean circuits. Finally, we also develop a novel implementation of gradient-based learning in Python, informed by the principles introduced by our framework.

* 14 pages  
View paper onarxiv icon

Share this with someone who'll enjoy it: