Do Multimodal LLMs See Sentiment?

Understanding how visual content communicates sentiment is critical in an era where online interaction is increasingly dominated by this kind of media on social platforms. However, this remains a challenging problem, as sentiment perception is closely tied to complex, scene-level semantics. In this paper, we propose an original framework, MLLMsent, to investigate the sentiment reasoning capabilities of Multimodal Large Language Models (MLLMs) through three perspectives: (1) using those MLLMs for direct sentiment classification from images; (2) associating them with pre-trained LLMs for sentiment analysis on automatically generated image descriptions; and (3) fine-tuning the LLMs on sentiment-labeled image descriptions. Experiments on a recent and established benchmark demonstrate that our proposal, particularly the fine-tuned approach, achieves state-of-the-art results outperforming Lexicon-, CNN-, and Transformer-based baselines by up to 30.9%, 64.8%, and 42.4%, respectively, across different levels of evaluators' agreement and sentiment polarity categories. Remarkably, in a cross-dataset test, without any training on these new data, our model still outperforms, by up to 8.26%, the best runner-up, which has been trained directly on them. These results highlight the potential of the proposed visual reasoning scheme for advancing affective computing, while also establishing new benchmarks for future research.

Computação Urbana da Teoria à Prática: Fundamentos, Aplicações e Desafios

The growing of cities has resulted in innumerable technical and managerial challenges for public administrators such as energy consumption, pollution, urban mobility and even supervision of private and public spaces in an appropriate way. Urban Computing emerges as a promising paradigm to solve such challenges, through the extraction of knowledge, from a large amount of heterogeneous data existing in urban space. Moreover, Urban Computing correlates urban sensing, data management, and analysis to provide services that have the potential to improve the quality of life of the citizens of large urban centers. Consider this context, this chapter aims to present the fundamentals of Urban Computing and the steps necessary to develop an application in this area. To achieve this goal, the following questions will be investigated, namely: (i) What are the main research problems of Urban Computing?; (ii) What are the technological challenges for the implementation of services in Urban Computing?; (iii) What are the main methodologies used for the development of services in Urban Computing?; and (iv) What are the representative applications in this field?

OutdoorSent: Sentiment Analysis of Urban Outdoor Images by Using Semantic and Deep Features

Opinion mining in outdoor images posted by users during day-to-day or leisure activities, for example, can provide valuable information to better understand urban areas. In this work, we propose a framework to classify the sentiment of outdoor images shared by users on social networks. We compare the performance of state-of-the-art ConvNet architectures, namely, VGG-16, Resnet50, and InceptionV3, as well as one specifically designed for sentiment analysis. The combination of such classifiers, a strategy known as ensemble, is also considered. We also use different experimental setups to evaluate how the merging of deep features and semantic information derived from the scene attributes can improve classification performance. The evaluation explores a novel dataset, namely OutdoorSent, of geolocalized urban outdoor images extracted from Instagram related to three sentiment polarities (positive, negative, and neutral), as well as another dataset publicly available (DeepSent). We observe that the incorporation of knowledge related to semantics features tend to improve the accuracy of low-complex ConvNet architectures. Furthermore, we also demonstrated the applicability of our results in the city of Chicago, United States, showing that they can help to understand the subjective characteristics of different areas of the city. For instance, particular areas of the city tend to concentrate more images of a specific class of sentiment. The ConvNet architectures, trained models, and the proposed outdoor image dataset will be publicly available at http://dainf.ct.utfpr.edu.br/outdoorsent.