A Randomized Permutation Whole-Model Test Heuristic for Self-Validated Ensemble Models (SVEM)

We introduce a heuristic to test the significance of fit of Self-Validated Ensemble Models (SVEM) against the null hypothesis of a constant response. A SVEM model averages predictions from nBoot fits of a model, applied to fractionally weighted bootstraps of the target dataset. It tunes each fit on a validation copy of the training data, utilizing anti-correlated weights for training and validation. The proposed test computes SVEM predictions centered by the response column mean and normalized by the ensemble variability at each of nPoint points spaced throughout the factor space. A reference distribution is constructed by refitting the SVEM model to nPerm randomized permutations of the response column and recording the corresponding standardized predictions at the nPoint points. A reduced-rank singular value decomposition applied to the centered and scaled nPerm x nPoint reference matrix is used to calculate the Mahalanobis distance for each of the nPerm permutation results as well as the jackknife (holdout) Mahalanobis distance of the original response column. The process is repeated independently for each response in the experiment, producing a joint graphical summary. We present a simulation driven power analysis and discuss limitations of the test relating to model flexibility and design adequacy. The test maintains the nominal Type I error rate even when the base SVEM model contains more parameters than observations.

Lessons Learned Applying Deep Learning Approaches to Forecasting Complex Seasonal Behavior

Deep learning methods have gained popularity in recent years through the media and the relative ease of implementation through open source packages such as Keras. We investigate the applicability of popular recurrent neural networks in forecasting call center volumes at a large financial services company. These series are highly complex with seasonal patterns - between hours of the day, day of the week, and time of the year - in addition to autocorrelation between individual observations. Though we investigate the financial services industry, the recommendations for modeling cyclical nonlinear behavior generalize across all sectors. We explore the optimization of parameter settings and convergence criteria for Elman (simple), Long Short-Term Memory (LTSM), and Gated Recurrent Unit (GRU) RNNs from a practical point of view. A designed experiment using actual call center data across many different "skills" (income call streams) compares performance measured by validation error rates of the best observed RNN configurations against other modern and classical forecasting techniques. We summarize the utility of and considerations required for using deep learning methods in forecasting.