Alert button
Picture for Shigang Li

Shigang Li

Alert button

TRANSOM: An Efficient Fault-Tolerant System for Training LLMs

Oct 18, 2023
Baodong Wu, Lei Xia, Qingping Li, Kangyu Li, Xu Chen, Yongqiang Guo, Tieyao Xiang, Yuheng Chen, Shigang Li

Viaarxiv icon

Co-design Hardware and Algorithm for Vector Search

Jul 06, 2023
Wenqi Jiang, Shigang Li, Yu Zhu, Johannes de Fine Licht, Zhenhao He, Runbin Shi, Cedric Renggli, Shuai Zhang, Theodoros Rekatsinas, Torsten Hoefler, Gustavo Alonso

Figure 1 for Co-design Hardware and Algorithm for Vector Search
Figure 2 for Co-design Hardware and Algorithm for Vector Search
Figure 3 for Co-design Hardware and Algorithm for Vector Search
Figure 4 for Co-design Hardware and Algorithm for Vector Search
Viaarxiv icon

ASDL: A Unified Interface for Gradient Preconditioning in PyTorch

May 08, 2023
Kazuki Osawa, Satoki Ishikawa, Rio Yokota, Shigang Li, Torsten Hoefler

Figure 1 for ASDL: A Unified Interface for Gradient Preconditioning in PyTorch
Figure 2 for ASDL: A Unified Interface for Gradient Preconditioning in PyTorch
Figure 3 for ASDL: A Unified Interface for Gradient Preconditioning in PyTorch
Figure 4 for ASDL: A Unified Interface for Gradient Preconditioning in PyTorch
Viaarxiv icon

An End-to-End Network for Upright Adjustment of Panoramic Images

Apr 12, 2023
Heyu Chen, Jianfeng Li, Shigang Li

Figure 1 for An End-to-End Network for Upright Adjustment of Panoramic Images
Figure 2 for An End-to-End Network for Upright Adjustment of Panoramic Images
Figure 3 for An End-to-End Network for Upright Adjustment of Panoramic Images
Figure 4 for An End-to-End Network for Upright Adjustment of Panoramic Images
Viaarxiv icon

PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices

Nov 25, 2022
Kazuki Osawa, Shigang Li, Torsten Hoefler

Figure 1 for PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices
Figure 2 for PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices
Figure 3 for PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices
Figure 4 for PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices
Viaarxiv icon

Efficient Quantized Sparse Matrix Operations on Tensor Cores

Sep 14, 2022
Shigang Li, Kazuki Osawa, Torsten Hoefler

Figure 1 for Efficient Quantized Sparse Matrix Operations on Tensor Cores
Figure 2 for Efficient Quantized Sparse Matrix Operations on Tensor Cores
Figure 3 for Efficient Quantized Sparse Matrix Operations on Tensor Cores
Figure 4 for Efficient Quantized Sparse Matrix Operations on Tensor Cores
Viaarxiv icon

HammingMesh: A Network Topology for Large-Scale Deep Learning

Sep 03, 2022
Torsten Hoefler, Tommaso Bonato, Daniele De Sensi, Salvatore Di Girolamo, Shigang Li, Marco Heddes, Jon Belk, Deepak Goel, Miguel Castro, Steve Scott

Figure 1 for HammingMesh: A Network Topology for Large-Scale Deep Learning
Figure 2 for HammingMesh: A Network Topology for Large-Scale Deep Learning
Figure 3 for HammingMesh: A Network Topology for Large-Scale Deep Learning
Figure 4 for HammingMesh: A Network Topology for Large-Scale Deep Learning
Viaarxiv icon

Near-Optimal Sparse Allreduce for Distributed Deep Learning

Jan 19, 2022
Shigang Li, Torsten Hoefler

Figure 1 for Near-Optimal Sparse Allreduce for Distributed Deep Learning
Figure 2 for Near-Optimal Sparse Allreduce for Distributed Deep Learning
Figure 3 for Near-Optimal Sparse Allreduce for Distributed Deep Learning
Figure 4 for Near-Optimal Sparse Allreduce for Distributed Deep Learning
Viaarxiv icon