Alert button
Picture for Less Wright

Less Wright

Alert button

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Apr 21, 2023
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Shen Li

Figure 1 for PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Figure 2 for PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Figure 3 for PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Figure 4 for PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Viaarxiv icon

Ranger21: a synergistic deep learning optimizer

Jun 25, 2021
Less Wright, Nestor Demeure

Figure 1 for Ranger21: a synergistic deep learning optimizer
Figure 2 for Ranger21: a synergistic deep learning optimizer
Figure 3 for Ranger21: a synergistic deep learning optimizer
Figure 4 for Ranger21: a synergistic deep learning optimizer
Viaarxiv icon