Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox


Training Multimodal Systems for Classification with Multiple Objectives

Aug 26, 2020
Jason Armitage, Shramana Thakur, Rishi Tripathi, Jens Lehmann, Maria Maleshkova


Share this with someone who'll enjoy it:


We learn about the world from a diverse range of sensory information. Automated systems lack this ability as investigation has centred on processing information presented in a single form. Adapting architectures to learn from multiple modalities creates the potential to learn rich representations of the world - but current multimodal systems only deliver marginal improvements on unimodal approaches. Neural networks learn sampling noise during training with the result that performance on unseen data is degraded. This research introduces a second objective over the multimodal fusion process learned with variational inference. Regularisation methods are implemented in the inner training loop to control variance and the modular structure stabilises performance as additional neurons are added to layers. This framework is evaluated on a multilabel classification task with textual and visual inputs to demonstrate the potential for multiple objectives and probabilistic methods to lower variance and improve generalisation.

* CLEOPATRA Workshop co-located with ESWC 2020 


   Access Paper Source



Share this with someone who'll enjoy it: