Alert button

Relationship-based Neural Baby Talk

Mar 08, 2021
Fan Fu, Tingting Xie, Ioannis Patras, Sepehr Jalali

Figure 1 for Relationship-based Neural Baby Talk
Figure 2 for Relationship-based Neural Baby Talk
Figure 3 for Relationship-based Neural Baby Talk
Figure 4 for Relationship-based Neural Baby Talk

Share this with someone who'll enjoy it:

Understanding interactions between objects in an image is an important element for generating captions. In this paper, we propose a relationship-based neural baby talk (R-NBT) model to comprehensively investigate several types of pairwise object interactions by encoding each image via three different relationship-based graph attention networks (GATs). We study three main relationships: \textit{spatial relationships} to explore geometric interactions, \textit{semantic relationships} to extract semantic interactions, and \textit{implicit relationships} to capture hidden information that could not be modelled explicitly as above. We construct three relationship graphs with the objects in an image as nodes, and the mutual relationships of pairwise objects as edges. By exploring features of neighbouring regions individually via GATs, we integrate different types of relationships into visual features of each node. Experiments on COCO dataset show that our proposed R-NBT model outperforms state-of-the-art models trained on COCO dataset in three image caption generation tasks.

View paper onarxiv icon

Share this with someone who'll enjoy it: