Human personality decides various aspects of their daily life and working behaviors. Since personality traits are relatively stable over time and unique for each subject, previous approaches frequently infer personality from a single frame or short-term behaviors. Moreover, most of them failed to specifically extract person-specific and unique cues for personality recognition. In this paper, we propose a novel video-based automatic personality traits recognition approach which consists of: (1) a \textbf{domain-specific facial behavior modelling} module that extracts personality-related multi-scale short-term human facial behavior features; (2) a \textbf{long-term behavior modelling} module that summarizes all short-term features of a video as a long-term/video-level personality representation and (3) a \textbf{multi-task personality traits prediction module} that models underlying relationship among all traits and jointly predict them based on the video-level personality representation. We conducted the experiments on ChaLearn First Impression dataset, and our approach achieved comparable results to the state-of-the-art. Importantly, we show that all three proposed modules brought important benefits for personality recognition.