Alert button

MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

Dec 19, 2022
Figure 1 for MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
Figure 2 for MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
Figure 3 for MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
Figure 4 for MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

Share this with someone who'll enjoy it:

We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously, towards high-quality realistic videos. To generate joint audio-video pairs, we propose a novel Multi-Modal Diffusion model (i.e., MM-Diffusion), with two-coupled denoising autoencoders. In contrast to existing single-modal diffusion models, MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Two subnets for audio and video learn to gradually generate aligned audio-video pairs from Gaussian noises. To ensure semantic consistency across modalities, we propose a novel random-shift based attention block bridging over the two subnets, which enables efficient cross-modal alignment, and thus reinforces the audio-video fidelity for each other. Extensive experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks (e.g., video-to-audio). In particular, we achieve the best FVD and FAD on Landscape and AIST++ dancing datasets. Turing tests of 10k votes further demonstrate dominant preferences for our model. The code and pre-trained models can be downloaded at https://github.com/researchmm/MM-Diffusion.

Share this with someone who'll enjoy it: