Linear programming-based solution methods for constrained POMDPs

Constrained partially observable Markov decision processes (CPOMDPs) have been used to model various real-world phenomena. However, they are notoriously difficult to solve to optimality, and there exist only a few approximation methods for obtaining high-quality solutions. In this study, we use grid-based approximations in combination with linear programming (LP) models to generate approximate policies for CPOMDPs. We consider five CPOMDP problem instances and conduct a detailed numerical study of both their finite and infinite horizon formulations. We first establish the quality of the approximate unconstrained POMDP policies through a comparative analysis with exact solution methods. We then show the performance of the LP-based CPOMDP solution approaches for varying budget levels (i.e., cost limits) for different problem instances. Finally, we show the flexibility of LP-based approaches by applying deterministic policy constraints, and investigate the impact that these constraints have on collected rewards and CPU run time. Our analysis demonstrates that LP models can effectively generate approximate policies for both finite and infinite horizon problems, while providing the flexibility to incorporate various additional constraints into the underlying model.

A multi-objective constrained POMDP model for breast cancer screening

Breast cancer is a common and deadly disease, but it is often curable when diagnosed early. While most countries have large-scale screening programs, there is no consensus on a single globally accepted policy for breast cancer screening. The complex nature of the disease; limited availability of screening methods such as mammography, magnetic resonance imaging (MRI), and ultrasound screening; and public health policies all factor into the development of screening policies. Resource availability concerns necessitate the design of policies which conform to a budget, a problem which can be modelled as a constrained partially observable Markov decision process (CPOMDP). In this study, we propose a multi-objective CPOMDP model for breast cancer screening with two objectives: minimize the lifetime risk of dying due to breast cancer and maximize the quality-adjusted life years. Additionally, we consider an expanded action space which allows for screening methods beyond mammography. Each action has a unique impact on quality-adjusted life years and lifetime risk, as well as a unique cost. Our results reveal the Pareto frontier of optimal solutions for average and high risk patients at different budget levels, which can be used by decision makers to set policies in practice.