This paper presents a Bayesian model selection approach via Bayesian quadrature and sensitivity analysis of the selection criterion for a lithium-ion battery model. The Bayesian model evidence is adopted as the metric, which can select the simplest but well-describing model based on Occam's razor principle. While the model evidence requires prohibitive integral computations over parameter space, Bayesian quadrature offers sample-efficient integration via model-based inference to minimise the number of battery model evaluations. The posterior distribution of battery model parameters can also be inferred as a byproduct in one go, which is also beneficial in creating a digital twin. The simplest lithium-ion battery models, equivalent circuit models, were used to analyse the sensitivity of the selection criterion at given different datasets and model configurations. We show that popular selection criteria, such as root-mean-square error, and Bayesian information criterion, can fail to select a correct model in a multimodal posterior case. The model evidence can spot the true model in such cases, simultaneously providing the variance of evidence inference itself as an indication of confidence. Bayesian quadrature can compute the evidence faster than popular MCMC solvers.