Traffic accidents cause over a million deaths every year, of which a large fraction is attributed to drunk driving. Automated drunk detection systems in vehicles are necessary to reduce traffic accidents and the related financial costs. Existing solutions require special equipment such as electrocardiogram, infrared cameras or breathalyzers. In this work, we propose a new dataset called DIF (Dataset of Intoxicated Faces) containing RGB face videos of drunk and sober people obtained from online sources. We analyze the face videos to extract features related to eye gaze, face pose and facial expressions. A recurrent neural network is used to model the evolution of these multimodal facial features. Our experiments show the eye gaze and facial expression features to be particularly discriminative for our dataset. We achieve good classification accuracy on the DIF dataset and show that face videos can be effectively used to detect drunk people. Such face videos can be readily acquired through a camera and used to prevent drunk driving incidents.