Alert button
Picture for Xiangjun Fan

Xiangjun Fan

Alert button

Deep Retrieval: An End-to-End Learnable Structure Model for Large-Scale Recommendations

Jul 12, 2020
Weihao Gao, Xiangjun Fan, Jiankai Sun, Kai Jia, Wenzhi Xiao, Chong Wang, Xiaobing Liu

Figure 1 for Deep Retrieval: An End-to-End Learnable Structure Model for Large-Scale Recommendations
Figure 2 for Deep Retrieval: An End-to-End Learnable Structure Model for Large-Scale Recommendations
Figure 3 for Deep Retrieval: An End-to-End Learnable Structure Model for Large-Scale Recommendations
Figure 4 for Deep Retrieval: An End-to-End Learnable Structure Model for Large-Scale Recommendations

One of the core problems in large-scale recommendations is to retrieve top relevant candidates accurately and efficiently, preferably in sub-linear time. Previous approaches are mostly based on a two-step procedure: first learn an inner-product model and then use maximum inner product search (MIPS) algorithms to search top candidates, leading to potential loss of retrieval accuracy. In this paper, we present Deep Retrieval (DR), an end-to-end learnable structure model for large-scale recommendations. DR encodes all candidates into a discrete latent space. Those latent codes for the candidates are model parameters and to be learnt together with other neural network parameters to maximize the same objective function. With the model learnt, a beam search over the latent codes is performed to retrieve the top candidates. Empirically, we showed that DR, with sub-linear computational complexity, can achieve almost the same accuracy as the brute-force baseline.

* 13 pages, 4 figures 
Viaarxiv icon