Alert button

Monolingual Recognizers Fusion for Code-switching Speech Recognition

Nov 02, 2022
Tongtong Song, Qiang Xu, Haoyu Lu, Longbiao Wang, Hao Shi, Yuqin Lin, Yanbing Yang, Jianwu Dang

Figure 1 for Monolingual Recognizers Fusion for Code-switching Speech Recognition
Figure 2 for Monolingual Recognizers Fusion for Code-switching Speech Recognition
Figure 3 for Monolingual Recognizers Fusion for Code-switching Speech Recognition
Figure 4 for Monolingual Recognizers Fusion for Code-switching Speech Recognition

Share this with someone who'll enjoy it:

The bi-encoder structure has been intensively investigated in code-switching (CS) automatic speech recognition (ASR). However, most existing methods require the structures of two monolingual ASR models (MAMs) should be the same and only use the encoder of MAMs. This leads to the problem that pre-trained MAMs cannot be timely and fully used for CS ASR. In this paper, we propose a monolingual recognizers fusion method for CS ASR. It has two stages: the speech awareness (SA) stage and the language fusion (LF) stage. In the SA stage, acoustic features are mapped to two language-specific predictions by two independent MAMs. To keep the MAMs focused on their own language, we further extend the language-aware training strategy for the MAMs. In the LF stage, the BELM fuses two language-specific predictions to get the final prediction. Moreover, we propose a text simulation strategy to simplify the training process of the BELM and reduce reliance on CS data. Experiments on a Mandarin-English corpus show the efficiency of the proposed method. The mix error rate is significantly reduced on the test set after using open-source pre-trained MAMs.

* Submitted to ICASSP2023  
View paper onarxiv icon

Share this with someone who'll enjoy it: