Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox
Picture for Di He

Di He

Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding


Jun 23, 2021
Shengjie Luo, Shanda Li, Tianle Cai, Di He, Dinglan Peng, Shuxin Zheng, Guolin Ke, Liwei Wang, Tie-Yan Liu

Figure 1 for Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Figure 2 for Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Figure 3 for Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Figure 4 for Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
* Preprint. Work in Progress 
  Access Paper or Ask Questions

First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track


Jun 20, 2021
Chengxuan Ying, Mingqi Yang, Shuxin Zheng, Guolin Ke, Shengjie Luo, Tianle Cai, Chenglin Wu, Yuxin Wang, Yanming Shen, Di He

Figure 1 for First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track
Figure 2 for First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track
Figure 3 for First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track
Figure 4 for First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track

  Access Paper or Ask Questions

Do Transformers Really Perform Bad for Graph Representation?


Jun 17, 2021
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu

Figure 1 for Do Transformers Really Perform Bad for Graph Representation?
Figure 2 for Do Transformers Really Perform Bad for Graph Representation?
Figure 3 for Do Transformers Really Perform Bad for Graph Representation?
Figure 4 for Do Transformers Really Perform Bad for Graph Representation?

  Access Paper or Ask Questions

How could Neural Networks understand Programs?


May 31, 2021
Dinglan Peng, Shuxin Zheng, Yatao Li, Guolin Ke, Di He, Tie-Yan Liu

Figure 1 for How could Neural Networks understand Programs?
Figure 2 for How could Neural Networks understand Programs?
Figure 3 for How could Neural Networks understand Programs?
Figure 4 for How could Neural Networks understand Programs?
* ICML 2021 
  Access Paper or Ask Questions

Adversarial Training with Rectified Rejection


May 31, 2021
Tianyu Pang, Huishuai Zhang, Di He, Yinpeng Dong, Hang Su, Wei Chen, Jun Zhu, Tie-Yan Liu

Figure 1 for Adversarial Training with Rectified Rejection
Figure 2 for Adversarial Training with Rectified Rejection
Figure 3 for Adversarial Training with Rectified Rejection
Figure 4 for Adversarial Training with Rectified Rejection

  Access Paper or Ask Questions

Wav2vec-C: A Self-supervised Model for Speech Representation Learning


Mar 09, 2021
Samik Sadhu, Di He, Che-Wei Huang, Sri Harish Mallidi, Minhua Wu, Ariya Rastrow, Andreas Stolcke, Jasha Droppo, Roland Maas

Figure 1 for Wav2vec-C: A Self-supervised Model for Speech Representation Learning
Figure 2 for Wav2vec-C: A Self-supervised Model for Speech Representation Learning
Figure 3 for Wav2vec-C: A Self-supervised Model for Speech Representation Learning
Figure 4 for Wav2vec-C: A Self-supervised Model for Speech Representation Learning

  Access Paper or Ask Questions

Transformers with Competitive Ensembles of Independent Mechanisms


Feb 27, 2021
Alex Lamb, Di He, Anirudh Goyal, Guolin Ke, Chien-Feng Liao, Mirco Ravanelli, Yoshua Bengio

Figure 1 for Transformers with Competitive Ensembles of Independent Mechanisms
Figure 2 for Transformers with Competitive Ensembles of Independent Mechanisms
Figure 3 for Transformers with Competitive Ensembles of Independent Mechanisms
Figure 4 for Transformers with Competitive Ensembles of Independent Mechanisms
* Under Review, ICML 2021 
  Access Paper or Ask Questions

LazyFormer: Self Attention with Lazy Update


Feb 25, 2021
Chengxuan Ying, Guolin Ke, Di He, Tie-Yan Liu

Figure 1 for LazyFormer: Self Attention with Lazy Update
Figure 2 for LazyFormer: Self Attention with Lazy Update
Figure 3 for LazyFormer: Self Attention with Lazy Update
Figure 4 for LazyFormer: Self Attention with Lazy Update

  Access Paper or Ask Questions

Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder


Feb 18, 2021
Shuqi Lu, Chenyan Xiong, Di He, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tieyan Liu, Arnold Overwijk

Figure 1 for Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder
Figure 2 for Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder
Figure 3 for Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder
Figure 4 for Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder

  Access Paper or Ask Questions

Revisiting Language Encoding in Learning Multilingual Representations


Feb 16, 2021
Shengjie Luo, Kaiyuan Gao, Shuxin Zheng, Guolin Ke, Di He, Liwei Wang, Tie-Yan Liu

Figure 1 for Revisiting Language Encoding in Learning Multilingual Representations
Figure 2 for Revisiting Language Encoding in Learning Multilingual Representations
Figure 3 for Revisiting Language Encoding in Learning Multilingual Representations
Figure 4 for Revisiting Language Encoding in Learning Multilingual Representations

  Access Paper or Ask Questions

Towards Certifying $\ell_\infty$ Robustness using Neural Networks with $\ell_\infty$-dist Neurons


Feb 10, 2021
Bohang Zhang, Tianle Cai, Zhou Lu, Di He, Liwei Wang

Figure 1 for Towards Certifying $\ell_\infty$ Robustness using Neural Networks with $\ell_\infty$-dist Neurons
Figure 2 for Towards Certifying $\ell_\infty$ Robustness using Neural Networks with $\ell_\infty$-dist Neurons
Figure 3 for Towards Certifying $\ell_\infty$ Robustness using Neural Networks with $\ell_\infty$-dist Neurons
Figure 4 for Towards Certifying $\ell_\infty$ Robustness using Neural Networks with $\ell_\infty$-dist Neurons

  Access Paper or Ask Questions

CODE-AE: A Coherent De-confounding Autoencoder for Predicting Patient-Specific Drug Response From Cell Line Transcriptomics


Jan 31, 2021
Di He, Lei Xie

Figure 1 for CODE-AE: A Coherent De-confounding Autoencoder for Predicting Patient-Specific Drug Response From Cell Line Transcriptomics
Figure 2 for CODE-AE: A Coherent De-confounding Autoencoder for Predicting Patient-Specific Drug Response From Cell Line Transcriptomics
Figure 3 for CODE-AE: A Coherent De-confounding Autoencoder for Predicting Patient-Specific Drug Response From Cell Line Transcriptomics
Figure 4 for CODE-AE: A Coherent De-confounding Autoencoder for Predicting Patient-Specific Drug Response From Cell Line Transcriptomics

  Access Paper or Ask Questions

A Cross-Level Information Transmission Network for Predicting Phenotype from New Genotype: Application to Cancer Precision Medicine


Oct 09, 2020
Di He, Lei Xie

Figure 1 for A Cross-Level Information Transmission Network for Predicting Phenotype from New Genotype: Application to Cancer Precision Medicine
Figure 2 for A Cross-Level Information Transmission Network for Predicting Phenotype from New Genotype: Application to Cancer Precision Medicine
Figure 3 for A Cross-Level Information Transmission Network for Predicting Phenotype from New Genotype: Application to Cancer Precision Medicine
Figure 4 for A Cross-Level Information Transmission Network for Predicting Phenotype from New Genotype: Application to Cancer Precision Medicine

  Access Paper or Ask Questions

GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training


Sep 07, 2020
Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Tie-yan Liu, Liwei Wang

Figure 1 for GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
Figure 2 for GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
Figure 3 for GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
Figure 4 for GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

  Access Paper or Ask Questions

Variance-reduced Language Pretraining via a Mask Proposal Network


Aug 12, 2020
Liang Chen, Tianyuan Zhang, Di He, Guolin Ke, Liwei Wang, Tie-Yan Liu

Figure 1 for Variance-reduced Language Pretraining via a Mask Proposal Network
Figure 2 for Variance-reduced Language Pretraining via a Mask Proposal Network
Figure 3 for Variance-reduced Language Pretraining via a Mask Proposal Network
Figure 4 for Variance-reduced Language Pretraining via a Mask Proposal Network

  Access Paper or Ask Questions

Taking Notes on the Fly Helps BERT Pre-training


Aug 04, 2020
Qiyu Wu, Chen Xing, Yatao Li, Guolin Ke, Di He, Tie-Yan Liu

Figure 1 for Taking Notes on the Fly Helps BERT Pre-training
Figure 2 for Taking Notes on the Fly Helps BERT Pre-training
Figure 3 for Taking Notes on the Fly Helps BERT Pre-training
Figure 4 for Taking Notes on the Fly Helps BERT Pre-training
* Qiyu Wu and Chen Xing contribute equally 
  Access Paper or Ask Questions

Transferred Discrepancy: Quantifying the Difference Between Representations


Jul 24, 2020
Yunzhen Feng, Runtian Zhai, Di He, Liwei Wang, Bin Dong

Figure 1 for Transferred Discrepancy: Quantifying the Difference Between Representations
Figure 2 for Transferred Discrepancy: Quantifying the Difference Between Representations
Figure 3 for Transferred Discrepancy: Quantifying the Difference Between Representations
Figure 4 for Transferred Discrepancy: Quantifying the Difference Between Representations
* 23 pages, 3 figures 
  Access Paper or Ask Questions

Rethinking Positional Encoding in Language Pre-training


Jul 09, 2020
Guolin Ke, Di He, Tie-Yan Liu

Figure 1 for Rethinking Positional Encoding in Language Pre-training
Figure 2 for Rethinking Positional Encoding in Language Pre-training
Figure 3 for Rethinking Positional Encoding in Language Pre-training
Figure 4 for Rethinking Positional Encoding in Language Pre-training
* refine writing; update down-stream results due to bug fix 
  Access Paper or Ask Questions

Rethinking the Positional Encoding in Language Pre-training


Jun 28, 2020
Guolin Ke, Di He, Tie-Yan Liu

Figure 1 for Rethinking the Positional Encoding in Language Pre-training
Figure 2 for Rethinking the Positional Encoding in Language Pre-training
Figure 3 for Rethinking the Positional Encoding in Language Pre-training
Figure 4 for Rethinking the Positional Encoding in Language Pre-training

  Access Paper or Ask Questions

MC-BERT: Efficient Language Pre-Training via a Meta Controller


Jun 16, 2020
Zhenhui Xu, Linyuan Gong, Guolin Ke, Di He, Shuxin Zheng, Liwei Wang, Jiang Bian, Tie-Yan Liu

Figure 1 for MC-BERT: Efficient Language Pre-Training via a Meta Controller
Figure 2 for MC-BERT: Efficient Language Pre-Training via a Meta Controller
Figure 3 for MC-BERT: Efficient Language Pre-Training via a Meta Controller
Figure 4 for MC-BERT: Efficient Language Pre-Training via a Meta Controller

  Access Paper or Ask Questions

Invertible Image Rescaling


May 12, 2020
Mingqing Xiao, Shuxin Zheng, Chang Liu, Yaolong Wang, Di He, Guolin Ke, Jiang Bian, Zhouchen Lin, Tie-Yan Liu

Figure 1 for Invertible Image Rescaling
Figure 2 for Invertible Image Rescaling
Figure 3 for Invertible Image Rescaling
Figure 4 for Invertible Image Rescaling

  Access Paper or Ask Questions

Incorporating BERT into Neural Machine Translation


Feb 17, 2020
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, Tie-Yan Liu

Figure 1 for Incorporating BERT into Neural Machine Translation
Figure 2 for Incorporating BERT into Neural Machine Translation
Figure 3 for Incorporating BERT into Neural Machine Translation
Figure 4 for Incorporating BERT into Neural Machine Translation
* Accepted to ICLR-2020 
  Access Paper or Ask Questions

MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius


Feb 15, 2020
Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsieh, Liwei Wang

Figure 1 for MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
Figure 2 for MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
Figure 3 for MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
Figure 4 for MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
* In ICLR 2020. 20 Pages 
  Access Paper or Ask Questions

On Layer Normalization in the Transformer Architecture


Feb 12, 2020
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, Tie-Yan Liu

Figure 1 for On Layer Normalization in the Transformer Architecture
Figure 2 for On Layer Normalization in the Transformer Architecture
Figure 3 for On Layer Normalization in the Transformer Architecture
Figure 4 for On Layer Normalization in the Transformer Architecture

  Access Paper or Ask Questions