Observation-Augmented Contextual Multi-Armed Bandits for Robotic Exploration with Uncertain Semantic Data

For robotic decision-making under uncertainty, the balance between exploitation and exploration of available options must be carefully taken into account. In this study, we introduce a new variant of contextual multi-armed bandits called observation-augmented CMABs (OA-CMABs) wherein a decision-making agent can utilize extra outcome observations from an external information source. CMABs model the expected option outcomes as a function of context features and hidden parameters, which are inferred from previous option outcomes. In OA-CMABs, external observations are also a function of context features and thus provide additional evidence about the hidden parameters. Yet, if an external information source is error-prone, the resulting posterior updates can harm decision-making performance unless the presence of errors is considered. To this end, we propose a robust Bayesian inference process for OA-CMABs that is based on the concept of probabilistic data validation. Our approach handles complex mixture model parameter priors and hybrid observation likelihoods for semantic data sources, allowing us to develop validation algorithms based on recently develop probabilistic semantic data association techniques. Furthermore, to more effectively cope with the combined sources of uncertainty in OA-CMABs, we derive a new active inference algorithm for option selection based on expected free energy minimization. This generalizes previous work on active inference for bandit-based robotic decision-making by accounting for faulty observations and non-Gaussian inference. Our approaches are demonstrated on a simulated asynchronous search site selection problem for space exploration. The results show that even if incorrect observations are provided by external information sources, efficient decision-making and robust parameter inference are still achieved in a wide variety of experimental conditions.

Active Inference for Autonomous Decision-Making with Contextual Multi-Armed Bandits

In autonomous robotic decision-making under uncertainty, the tradeoff between exploitation and exploration of available options must be considered. If secondary information associated with options can be utilized, such decision-making problems can often be formulated as a contextual multi-armed bandits (CMABs). In this study, we apply active inference, which has been actively studied in the field of neuroscience in recent years, as an alternative action selection strategy for CMABs. Unlike conventional action selection strategies, it is possible to rigorously evaluate the uncertainty of each option when calculating the expected free energy (EFE) associated with the decision agent's probabilistic model, as derived from the free-energy principle. We specifically address the case where a categorical observation likelihood function is used, such that EFE values are analytically intractable. We introduce new approximation methods for computing the EFE based on variational and Laplace approximations. Extensive simulation study results demonstrate that, compared to other strategies, active inference generally requires far fewer iterations to identify optimal options and generally achieves superior cumulative regret, for relatively low extra computational cost.

Probabilistic Semantic Data Association for Collaborative Human-Robot Sensing

In collaborative human-robot semantic sensing problems, e.g. for scientific exploration, robots could potentially overtrust information given by a human partner, resulting in suboptimal state estimation and poor team performance. When humans cannot be treated as oracles, robots need to update state beliefs to correctly account for possible discrepancies between human semantic observations and the actual world states which lead to those observations. This work develops strategies for rigorous online calculation of probabilistic semantic data association (PSDA) probabilities for semantic likelihoods in general settings, unlike previous work which developed naive or heuristic approximations for specific settings. The new PSDA method is incorporated into a hybrid Bayesian data fusion scheme which uses Gaussian mixture priors for object states and softmax functions for semantic human sensor observation likelihoods, and is demonstrated in Monte Carlo simulations of collaborative multi-object search missions featuring a range of relevant human sensing characteristics (e.g. false detection rate). It is shown that PSDA leads to robust estimation of observation association probabilities under a wide range of conditions whenever semantic human sensor data contain significant target reference ambiguities for autonomous object search and localization.