Alert button
Picture for Daniele De Sensi

Daniele De Sensi

Alert button

HammingMesh: A Network Topology for Large-Scale Deep Learning

Sep 03, 2022
Torsten Hoefler, Tommaso Bonato, Daniele De Sensi, Salvatore Di Girolamo, Shigang Li, Marco Heddes, Jon Belk, Deepak Goel, Miguel Castro, Steve Scott

Figure 1 for HammingMesh: A Network Topology for Large-Scale Deep Learning
Figure 2 for HammingMesh: A Network Topology for Large-Scale Deep Learning
Figure 3 for HammingMesh: A Network Topology for Large-Scale Deep Learning
Figure 4 for HammingMesh: A Network Topology for Large-Scale Deep Learning

Numerous microarchitectural optimizations unlocked tremendous processing power for deep neural networks that in turn fueled the AI revolution. With the exhaustion of such optimizations, the growth of modern AI is now gated by the performance of training systems, especially their data movement. Instead of focusing on single accelerators, we investigate data-movement characteristics of large-scale training at full system scale. Based on our workload analysis, we design HammingMesh, a novel network topology that provides high bandwidth at low cost with high job scheduling flexibility. Specifically, HammingMesh can support full bandwidth and isolation to deep learning training jobs with two dimensions of parallelism. Furthermore, it also supports high global bandwidth for generic traffic. Thus, HammingMesh will power future large-scale deep learning systems with extreme bandwidth requirements.

* to appear at ACM/IEEE Supercomputing (SC22) 
Viaarxiv icon