Alert button

On Embeddings for Numerical Features in Tabular Deep Learning

Mar 10, 2022
Yura Gorishniy, Ivan Rubachev, Artem Babenko

Figure 1 for On Embeddings for Numerical Features in Tabular Deep Learning
Figure 2 for On Embeddings for Numerical Features in Tabular Deep Learning
Figure 3 for On Embeddings for Numerical Features in Tabular Deep Learning
Figure 4 for On Embeddings for Numerical Features in Tabular Deep Learning

Share this with someone who'll enjoy it:

Recently, Transformer-like deep architectures have shown strong performance on tabular data problems. Unlike traditional models, e.g., MLP, these architectures map scalar values of numerical features to high-dimensional embeddings before mixing them in the main backbone. In this work, we argue that embeddings for numerical features are an underexplored degree of freedom in tabular DL, which allows constructing more powerful DL models and competing with GBDT on some traditionally GBDT-friendly benchmarks. We start by describing two conceptually different approaches to building embedding modules: the first one is based on a piecewise linear encoding of scalar values, and the second one utilizes periodic activations. Then, we empirically demonstrate that these two approaches can lead to significant performance boosts compared to the embeddings based on conventional blocks such as linear layers and ReLU activations. Importantly, we also show that embedding numerical features is beneficial for many backbones, not only for Transformers. Specifically, after proper embeddings, simple MLP-like models can perform on par with the attention-based architectures. Overall, we highlight that embeddings for numerical features are an important design aspect, which has good potential for further improvements in tabular DL.

* Code: https://github.com/Yura52/rtdl  
View paper onarxiv iconopen_review iconOpenReview

Share this with someone who'll enjoy it: