Action detection and public traffic safety are crucial aspects of a safe community and a better society. Monitoring traffic flows in a smart city using different surveillance cameras can play a significant role in recognizing accidents and alerting first responders. The utilization of action recognition (AR) in computer vision tasks has contributed towards high-precision applications in video surveillance, medical imaging, and digital signal processing. This paper presents an intensive review focusing on action recognition in accident detection and autonomous transportation systems for a smart city. In this paper, we focused on AR systems that used diverse sources of traffic video capturing, such as static surveillance cameras on traffic intersections, highway monitoring cameras, drone cameras, and dash-cams. Through this review, we identified the primary techniques, taxonomies, and algorithms used in AR for autonomous transportation and accident detection. We also examined data sets utilized in the AR tasks, identifying the main sources of datasets and features of the datasets. This paper provides potential research direction to develop and integrate accident detection systems for autonomous cars and public traffic safety systems by alerting emergency personnel and law enforcement in the event of road accidents to minimize human error in accident reporting and provide a spontaneous response to victims