Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters

Machine learning methods are being increasingly used in most technical areas such as image recognition, product recommendation, financial analysis, medical diagnosis, and predictive maintenance. The key question that arises is: how do we control the learning process according to our requirement for the problem? Hyperparameter tuning is used to choose the optimal set of hyperparameters for controlling the learning process of a model. Selecting the appropriate hyperparameters directly impacts the performance measure a model. We have used simulation optimization using discrete search methods like ranking and selection (R&S) methods such as the KN method and stochastic ruler method and its variations for hyperparameter optimization and also developed the theoretical basis for applying common R&S methods. The KN method finds the best possible system with statistical guarantee and stochastic ruler method asymptotically converges to the optimal solution and is also computationally very efficient. We also benchmarked our results with state of art hyperparameter optimization libraries such as $hyperopt$ and $mango$ and found KN and stochastic ruler to be performing consistently better than $hyperopt~rand$ and stochastic ruler to be equally efficient in comparison with $hyperopt~tpe$ in most cases, even when our computational implementations are not yet optimized in comparison to professional packages.