Alert button

Using Active Speaker Faces for Diarization in TV shows

Mar 30, 2022
Rahul Sharma, Shrikanth Narayanan

Figure 1 for Using Active Speaker Faces for Diarization in TV shows
Figure 2 for Using Active Speaker Faces for Diarization in TV shows
Figure 3 for Using Active Speaker Faces for Diarization in TV shows
Figure 4 for Using Active Speaker Faces for Diarization in TV shows

Share this with someone who'll enjoy it:

Speaker diarization is one of the critical components of computational media intelligence as it enables a character-level analysis of story portrayals and media content understanding. Automated audio-based speaker diarization of entertainment media poses challenges due to the diverse acoustic conditions present in media content, be it background music, overlapping speakers, or sound effects. At the same time, speaking faces in the visual modality provide complementary information and not prone to the errors seen in the audio modality. In this paper, we address the problem of speaker diarization in TV shows using the active speaker faces. We perform face clustering on the active speaker faces and show superior speaker diarization performance compared to the state-of-the-art audio-based diarization methods. We additionally report a systematic analysis of the impact of active speaker face detection quality on the diarization performance. We also observe that a moderately well-performing active speaker system could outperform the audio-based diarization systems.

* Submitted to Interspeech 2022  
View paper onarxiv icon

Share this with someone who'll enjoy it: