Fall event detection, as one of the greatest risks to the elderly, has been a hot research issue in the solitary scene in recent years. Nevertheless, there are few researches on the fall event detection in complex background. Different from most conventional background subtraction methods which depend on background modeling, Mask R-CNN method based on deep learning technique can clearly extract the moving object in noise background. We further propose an attention guided Bi-directional LSTM model for the final fall event detection. To demonstrate the efficiency, the proposed method is verified in the public dataset and self-build dataset. Evaluation of the algorithm performances in comparison with other state-of-the-art methods indicates that the proposed design is accurate and robust, which means it is suitable for the task of fall event detection in complex situation.