Alert button

Multimodal Event Graphs: Towards Event Centric Understanding of Multimodal World

Jun 14, 2022
Hammad A. Ayyubi, Christopher Thomas, Lovish Chum, Rahul Lokesh, Yulei Niu, Xudong Lin, Long Chen, Jaywon Koo, Sounak Ray, Shih-Fu Chang

Figure 1 for Multimodal Event Graphs: Towards Event Centric Understanding of Multimodal World
Figure 2 for Multimodal Event Graphs: Towards Event Centric Understanding of Multimodal World
Figure 3 for Multimodal Event Graphs: Towards Event Centric Understanding of Multimodal World
Figure 4 for Multimodal Event Graphs: Towards Event Centric Understanding of Multimodal World

Share this with someone who'll enjoy it:

Understanding how events described or shown in multimedia content relate to one another is a critical component to developing robust artificially intelligent systems which can reason about real-world media. While much research has been devoted to event understanding in the text, image, and video domains, none have explored the complex relations that events experience across domains. For example, a news article may describe a `protest' event while a video shows an `arrest' event. Recognizing that the visual `arrest' event is a subevent of the broader `protest' event is a challenging, yet important problem that prior work has not explored. In this paper, we propose the novel task of MultiModal Event Event Relations to recognize such cross-modal event relations. We contribute a large-scale dataset consisting of 100k video-news article pairs, as well as a benchmark of densely annotated data. We also propose a weakly supervised multimodal method which integrates commonsense knowledge from an external knowledge base (KB) to predict rich multimodal event hierarchies. Experiments show that our model outperforms a number of competitive baselines on our proposed benchmark. We also perform a detailed analysis of our model's performance and suggest directions for future research.

View paper onarxiv icon

Share this with someone who'll enjoy it: