Alert button
Picture for Payam Mousavi

Payam Mousavi

Alert button

MaskRenderer: 3D-Infused Multi-Mask Realistic Face Reenactment

Sep 10, 2023
Tina Behrouzi, Atefeh Shahroudnejad, Payam Mousavi

Figure 1 for MaskRenderer: 3D-Infused Multi-Mask Realistic Face Reenactment
Figure 2 for MaskRenderer: 3D-Infused Multi-Mask Realistic Face Reenactment
Figure 3 for MaskRenderer: 3D-Infused Multi-Mask Realistic Face Reenactment
Figure 4 for MaskRenderer: 3D-Infused Multi-Mask Realistic Face Reenactment

We present a novel end-to-end identity-agnostic face reenactment system, MaskRenderer, that can generate realistic, high fidelity frames in real-time. Although recent face reenactment works have shown promising results, there are still significant challenges such as identity leakage and imitating mouth movements, especially for large pose changes and occluded faces. MaskRenderer tackles these problems by using (i) a 3DMM to model 3D face structure to better handle pose changes, occlusion, and mouth movements compared to 2D representations; (ii) a triplet loss function to embed the cross-reenactment during training for better identity preservation; and (iii) multi-scale occlusion, improving inpainting and restoring missing areas. Comprehensive quantitative and qualitative experiments conducted on the VoxCeleb1 test set, demonstrate that MaskRenderer outperforms state-of-the-art models on unseen faces, especially when the Source and Driving identities are very different.

Viaarxiv icon