Stronger Graph Transformer with Regularized Attention Scores

Graph Neural Networks are notorious for its memory consumption. A recent Transformer based GNN called Graph Transformer are shown to obtain superior performances when long range dependencies exist. However, combining graph data and Transformer architecture led to a combinationally worse memory issue. We propose a novel version of "edge regularization technique" that alleviates the need for Positional Encoding and ultimately alleviate GT's out of memory issue. We observe that it is not clear whether having an edge regularization on top of positional encoding is helpful. However, it seems evident when no positional encoding is applied, edge regularization technique indeed stably improves GT's performance.