Alert button

READ Avatars: Realistic Emotion-controllable Audio Driven Avatars

Mar 01, 2023
Jack Saunders, Vinay Namboodiri

Figure 1 for READ Avatars: Realistic Emotion-controllable Audio Driven Avatars
Figure 2 for READ Avatars: Realistic Emotion-controllable Audio Driven Avatars
Figure 3 for READ Avatars: Realistic Emotion-controllable Audio Driven Avatars
Figure 4 for READ Avatars: Realistic Emotion-controllable Audio Driven Avatars

Share this with someone who'll enjoy it:

We present READ Avatars, a 3D-based approach for generating 2D avatars that are driven by audio input with direct and granular control over the emotion. Previous methods are unable to achieve realistic animation due to the many-to-many nature of audio to expression mappings. We alleviate this issue by introducing an adversarial loss in the audio-to-expression generation process. This removes the smoothing effect of regression-based models and helps to improve the realism and expressiveness of the generated avatars. We note furthermore, that audio should be directly utilized when generating mouth interiors and that other 3D-based methods do not attempt this. We address this with audio-conditioned neural textures, which are resolution-independent. To evaluate the performance of our method, we perform quantitative and qualitative experiments, including a user study. We also propose a new metric for comparing how well an actor's emotion is reconstructed in the generated avatar. Our results show that our approach outperforms state of the art audio-driven avatar generation methods across several metrics. A demo video can be found at \url{https://youtu.be/QSyMl3vV0pA}

* 13 Pages, 8 Figures For demo video see https://youtu.be/QSyMl3vV0pA  
View paper onarxiv icon

Share this with someone who'll enjoy it: