Alert button

Multi-modal Automated Speech Scoring using Attention Fusion

May 17, 2020
Manraj Singh Grover, Yaman Kumar, Sumit Sarin, Payman Vafaee, Mika Hama, Rajiv Ratn Shah

Figure 1 for Multi-modal Automated Speech Scoring using Attention Fusion
Figure 2 for Multi-modal Automated Speech Scoring using Attention Fusion
Figure 3 for Multi-modal Automated Speech Scoring using Attention Fusion
Figure 4 for Multi-modal Automated Speech Scoring using Attention Fusion

Share this with someone who'll enjoy it:

In this study, we propose a novel multi-modal end-to-end neural approach for automated assessment of non-native English speakers' spontaneous speech using attention fusion. The pipeline employs Bi-directional Recurrent Convolutional Neural Networks and Bi-directional Long Short-Term Memory Neural Networks to encode acoustic and lexical cues from spectrograms and transcriptions, respectively. Attention fusion is performed on these learned predictive features to learn complex interactions between different modalities before final scoring. We compare our model with strong baselines and find combined attention to both lexical and acoustic cues significantly improves the overall performance of the system. Further, we present a qualitative and quantitative analysis of our model.

* Submitted to INTERSPEECH 2020  
View paper onarxiv icon

Share this with someone who'll enjoy it: