In this work, our aim is to identify whether the choice of virtualization strategy influences the performance of simulations in robotics. Performance is quantified in the error between a reference trajectory and the actual trajectory for the ball moving along the surface of a smooth plate. The two-sample Kolmogorov-Smirnov test is used to assess significance of variations in performance under the different experimental settings. Our results show that the selection of virtualization technology does have a significant effect on simulation, and moreover this effect can be amplified by the use of some operating systems. While these results are a strong cause for caution, they also provide reason for optimism for those considering 'repeatable robotics research' using virtualization.