Picture for Yingyu Liang

Yingyu Liang

Circuit Complexity Bounds for RoPE-based Transformer Architecture

Add code
Nov 12, 2024
Viaarxiv icon

Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study

Add code
Oct 15, 2024
Figure 1 for Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study
Figure 2 for Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study
Figure 3 for Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study
Viaarxiv icon

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix

Add code
Oct 15, 2024
Viaarxiv icon

Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent

Add code
Oct 15, 2024
Viaarxiv icon

HSR-Enhanced Sparse Attention Acceleration

Add code
Oct 14, 2024
Figure 1 for HSR-Enhanced Sparse Attention Acceleration
Figure 2 for HSR-Enhanced Sparse Attention Acceleration
Viaarxiv icon

Looped ReLU MLPs May Be All You Need as Practical Programmable Computers

Add code
Oct 12, 2024
Viaarxiv icon

Fine-grained Attention I/O Complexity: Comprehensive Analysis for Backward Passes

Add code
Oct 12, 2024
Viaarxiv icon

Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction

Add code
Sep 25, 2024
Figure 1 for Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
Figure 2 for Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
Figure 3 for Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
Figure 4 for Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
Viaarxiv icon

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time

Add code
Aug 23, 2024
Viaarxiv icon

A Tighter Complexity Analysis of SparseGPT

Add code
Aug 22, 2024
Viaarxiv icon