Advanced Driver Assistance Systems (ADAS) improve driving safety significantly. They alert drivers from unsafe traffic conditions when a dangerous maneuver appears. Traditional methods to predict driving maneuvers are mostly based on data-driven models alone. However, existing methods to understand the driver's intention remain an ongoing challenge due to a lack of intersection of human cognition and data analysis. To overcome this challenge, we propose a novel method that combines both the cognition-driven model and the data-driven model. We introduce a model named Cognitive Fusion-RNN (CF-RNN) which fuses the data inside the vehicle and the data outside the vehicle in a cognitive way. The CF-RNN model consists of two Long Short-Term Memory (LSTM) branches regulated by human reaction time. Experiments on the Brain4Cars benchmark dataset demonstrate that the proposed method outperforms previous methods and achieves state-of-the-art performance.