This paper introduces the R package ForecastTB that can be used to compare the forecasting accuracy of different methods as related to the characteristics of a dataset. The ForecastTB is a plug-and-play structured module, and several forecasting methods can be included with simple instructions. The proposed test-bench is not limited to the default forecasting and error metric functions, and users are able to append, remove or choose the desired methods as per requirements. Besides, several plotting functions are provided to allow comparative visualization of the performance and behavior of different forecasting methods. Furthermore, this paper presents example applications with natural time series datasets to exhibit the feature of the ForecastTB package to evaluate forecasting comparison analysis as affected by characteristics of a dataset.