ET-USB: Transformer-Based Sequential Behavior Modeling for Inbound Customer Service

Deep learning models with attention mechanisms have achieved exceptional results for many tasks, including language tasks and recommendation systems. Whereas previous studies have emphasized allocation of phone agents, we focused on inbound call prediction for customer service. A common method of analyzing user history behaviors is to extract all types of aggregated feature over time, but that method may fail to detect users' behavioral sequences. Therefore, we created a new approach, ET-USB, that incorporates users' sequential and nonsequential features; we apply the powerful Transformer encoder, a self-attention network model, to capture the information underlying user behavior sequences. ET-USB is helpful in various business scenarios at Cathay Financial Holdings. We conducted experiments to test the proposed network structure's ability to process various dimensions of behavior data; the results suggest that ET-USB delivers results superior to those of delivered by other deep-learning models.

SesameBERT: Attention for Anywhere

Fine-tuning with pre-trained models has achieved exceptional results for many language tasks. In this study, we focused on one such self-attention network model, namely BERT, which has performed well in terms of stacking layers across diverse language-understanding benchmarks. However, in many downstream tasks, information between layers is ignored by BERT for fine-tuning. In addition, although self-attention networks are well-known for their ability to capture global dependencies, room for improvement remains in terms of emphasizing the importance of local contexts. In light of these advantages and disadvantages, this paper proposes SesameBERT, a generalized fine-tuning method that (1) enables the extraction of global information among all layers through Squeeze and Excitation and (2) enriches local information by capturing neighboring contexts via Gaussian blurring. Furthermore, we demonstrated the effectiveness of our approach in the HANS dataset, which is used to determine whether models have adopted shallow heuristics instead of learning underlying generalizations. The experiments revealed that SesameBERT outperformed BERT with respect to GLUE benchmark and the HANS evaluation set.