This paper presents a novel dataset for traffic accidents analysis.Our goal is to resolve the lack of public data for research about automatic spatio-temporal annotations for traffic safety in the roads. Our Car Accident Detection and Prediction(CADP) dataset consists of 1,416 video segments collected from YouTube, with 205 video segments having full spatio-temporal annotations. To the best of our knowledge, our dataset is largest in terms of number of traffic accidents, compared to related datasets. Through the analysis of the proposed dataset, we observed a significant degradation of object detection in pedestrian category in our dataset, due to the object sizes and complexity of the scenes. To this end, we propose to integrate contextual information into conventional Faster R-CNN using Context Mining(CM) and Augmented Context Mining(ACM) to complement the accuracy for small pedestrian detection. Our experiments indicate a considerable improvement in object detection accuracy: +8.51 % for CM and +6.20 % for ACM. For person(pedestrian) category, we observed significant improvements:+46.45 % for CM and 45.22 % for ACM, compared to Faster R-CNN. Finally, we demonstrate the performance of accident forecasting in our dataset using Faster R-CNN and an Accident LSTM architecture. We achieved an average of 1.359 seconds in terms of Time-To-Accident measure with an Average Precision of 47.36 %. Our Webpage for the paper is https://goo.gl/cqK2wE