A Statistical Analysis for Per-Instance Evaluation of Stochastic Optimizers: How Many Repeats Are Enough?

A key trait of stochastic optimizers is that multiple runs of the same optimizer in attempting to solve the same problem can produce different results. As a result, their performance is evaluated over several repeats, or runs, on the problem. However, the accuracy of the estimated performance metrics depends on the number of runs and should be studied using statistical tools. We present a statistical analysis of the common metrics, and develop guidelines for experiment design to measure the optimizer's performance using these metrics to a high level of confidence and accuracy. To this end, we first discuss the confidence interval of the metrics and how they are related to the number of runs of an experiment. We then derive a lower bound on the number of repeats in order to guarantee achieving a given accuracy in the metrics. Using this bound, we propose an algorithm to adaptively adjust the number of repeats needed to ensure the accuracy of the evaluated metric. Our simulation results demonstrate the utility of our analysis and how it allows us to conduct reliable benchmarking as well as hyperparameter tuning and prevent us from drawing premature conclusions regarding the performance of stochastic optimizers.

Quantum Multiple Kernel Learning

Kernel methods play an important role in machine learning applications due to their conceptual simplicity and superior performance on numerous machine learning tasks. Expressivity of a machine learning model, referring to the ability of the model to approximate complex functions, has a significant influence on its performance in these tasks. One approach to enhancing the expressivity of kernel machines is to combine multiple individual kernels to arrive at a more expressive combined kernel. This approach is referred to as multiple kernel learning (MKL). In this work, we propose an MKL method we refer to as quantum MKL, which combines multiple quantum kernels. Our method leverages the power of deterministic quantum computing with one qubit (DQC1) to estimate the combined kernel for a set of classically intractable individual quantum kernels. The combined kernel estimation is achieved without explicitly computing each individual kernel, while still allowing for the tuning of individual kernels in order to achieve better expressivity. Our simulations on two binary classification problems---one performed on a synthetic dataset and the other on a German credit dataset---demonstrate the superiority of the quantum MKL method over single quantum kernel machines.