Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Feb 01, 2020
Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, Cheng Niu, Jie Zhou

Share this with someone who'll enjoy it:

Audio-Visual Scene-Aware Dialog (AVSD) is a task to generate responses when chatting about a given video, which is organized as a track of the 8th Dialog System Technology Challenge (DSTC8). To solve the task, we propose a universal multimodal transformer and introduce the multi-task learning method to learn joint representations among different modalities as well as generate informative and fluent responses. Our method extends the natural language generation pre-trained model to multimodal dialogue generation task. Our system achieves the best performance in both objective and subjective evaluations in the challenge.

* Accepted by AAAI2020 DSTC8 workshop. Ranked 1st in DSTC8-AVSD track 

   Access Paper Source

Share this with someone who'll enjoy it: