Optimization meets Big Data: A survey

This paper reviews recent advances in big data optimization, providing the state-of-art of this emerging field. The main focus in this review are optimization techniques being applied in big data analysis environments. Integer linear programming, coordinate descent methods, alternating direction method of multipliers, simulation optimization and metaheuristics like evolutionary and genetic algorithms, particle swarm optimization, differential evolution, fireworks, bat, firefly and cuckoo search algorithms implementations are reviewed and discussed. The relation between big data optimization and software engineering topics like information work-flow styles, software architectures, and software framework is discussed. Comparative analysis in platforms being used in big data optimization environments are highlighted in order to bring a state-or-art of possible architectures and topologies.

Machine learning for improving performance in an evolutionary algorithm for minimum path with uncertain costs given by massively simulated scenarios

In this work we introduce an implementation for which machine learning techniques helped improve the overall performance of an evolutionary algorithm for an optimization problem, namely a variation of robust minimum-cost path in graphs. In this big data optimization problem, a path achieving a good cost in most scenarios from an available set of scenarios (generated by a simulation process) must be obtained. The most expensive task of our evolutionary algorithm, in terms of computational resources, is the evaluation of candidate paths: the fitness function must calculate the cost of the candidate path in every generated scenario. Given the large number of scenarios, this task must be implemented in a distributed environment. We implemented gradient boosting decision trees to classify candidate paths in order to identify good candidates. The cost of the not-so-good candidates is simply forecasted. We studied the training process, gain performance, accuracy, and other variables. Our computational experiments show that the computational performance was significantly improved at the expense of a limited loss of accuracy.