Alert button

Position Information in Transformers: An Overview

Feb 22, 2021
Philipp Dufter, Martin Schmitt, Hinrich Schütze

Figure 1 for Position Information in Transformers: An Overview
Figure 2 for Position Information in Transformers: An Overview
Figure 3 for Position Information in Transformers: An Overview
Figure 4 for Position Information in Transformers: An Overview

Share this with someone who'll enjoy it:

Transformers are arguably the main workhorse in recent Natural Language Processing research. By definition a Transformer is invariant with respect to reorderings of the input. However, language is inherently sequential and word order is essential to the semantics and syntax of an utterance. In this paper, we provide an overview of common methods to incorporate position information into Transformer models. The objectives of this survey are to i) showcase that position information in Transformer is a vibrant and extensive research area; ii) enable the reader to compare existing methods by providing a unified notation and meaningful clustering; iii) indicate what characteristics of an application should be taken into account when selecting a position encoding; iv) provide stimuli for future research.

View paper onarxiv icon

Share this with someone who'll enjoy it: