AI Overviews in Academic Search: Evaluating AI-generated Summaries of Search Results in a Domain-specific Search Engine

Evaluating search engine results pages (SERPs) to assess result relevance is a demanding step in academic search. In a formative mixed-methods design study, we examine AI-generated SERP-level summaries as a support feature in an academic search engine for social science information. First, we manually evaluated summaries of the top five results for 10 queries using two general-purpose models, one commercial and one open, deriving an exploratory six-category error taxonomy and five safeguards for scholarly deployment. We then conducted a within-subjects user study (n = 30) comparing interfaces with and without AI summaries. Confirmatory analyses showed consistent but non-significant trends favoring AI summaries for subjective workload, perceived usefulness, satisfaction, and decision-making confidence. Exploratory analyses suggested lower mental demand, with frustration also tending to be lower. Behaviorally, participants rarely expanded the summaries and descriptively made slightly fewer result clicks and query reformulations when summaries were available. Drawing on Information Foraging Theory and participant feedback, we suggest that AI summaries may concentrate SERP-level information scent to support early triage. Overall, the findings indicate that SERP-level AI summaries are a context- and user-dependent aid rather than a universal improvement, while contributing an error taxonomy, safeguard-aware deployment guidance, and concrete design implications for scholarly search.

Information Need in Metaverse Recordings -- A Field Study

Metaverse Recordings (MVRs) represent an emerging and underexplored media type within the field of Multimedia Information Retrieval (MMIR). This paper presents findings from a field study aimed at understanding the users information needs and search behaviors specific to MVR retrieval. By conducting and analyzing expert interviews, the study identifies application scenarios and highlights challenges in retrieving multimedia content from the metaverse. The results reveal existing application scenarios of MVRs and confirm the relevance of capturing time-series data from the graphical rendering process and related input-output devices, which are also highly relevant to user needs. Furthermore, the study provides a foundation for developing retrieval systems tailored to MVRs by defining use cases, user stereotypes, and specific requirements for MVR Retrieval systems. The findings contribute to a better understanding of information search behaviors in MVR Retrieval and pave the way for future research and system design in this field.

Report on the Workshop on Simulations for Information Access (Sim4IA 2024) at SIGIR 2024

This paper is a report of the Workshop on Simulations for Information Access (Sim4IA) workshop at SIGIR 2024. The workshop had two keynotes, a panel discussion, nine lightning talks, and two breakout sessions. Key takeaways were user simulation's importance in academia and industry, the possible bridging of online and offline evaluation, and the issues of organizing a companion shared task around user simulations for information access. We report on how we organized the workshop, provide a brief overview of what happened at the workshop, and summarize the main topics and findings of the workshop and future work.

Reinforcement Learning-driven Information Seeking: A Quantum Probabilistic Approach

Understanding an information forager's actions during interaction is very important for the study of interactive information retrieval. Although information spread in uncertain information space is substantially complex due to the high entanglement of users interacting with information objects~(text, image, etc.). However, an information forager, in general, accompanies a piece of information (information diet) while searching (or foraging) alternative contents, typically subject to decisive uncertainty. Such types of uncertainty are analogous to measurements in quantum mechanics which follow the uncertainty principle. In this paper, we discuss information seeking as a reinforcement learning task. We then present a reinforcement learning-based framework to model forager exploration that treats the information forager as an agent to guide their behaviour. Also, our framework incorporates the inherent uncertainty of the foragers' action using the mathematical formalism of quantum mechanics.