In traffic management, it is a very important issue to shorten the response time by detecting the incidents (accident, vehicle breakdown, an object falling on the road, etc.) and informing the corresponding personnel. In this study, an anomaly detection framework for road junctions is proposed. The final judgment is based on the trajectories followed by the vehicles. Trajectory information is provided by vehicle detection and tracking algorithms on visual data streamed from a fisheye camera. Deep learning algorithms are used for vehicle detection, and Kalman Filter is used for tracking. To observe the trajectories more accurately, the detected vehicle coordinates are transferred to the bird's eye view coordinates using the lens distortion model prediction algorithm. The system determines whether there is an abnormality in trajectories by comparing historical trajectory data and instantaneous incoming data. The proposed system has achieved 84.6% success in vehicle detection and 96.8% success in abnormality detection on synthetic data. The system also works with a 97.3% success rate in detecting abnormalities on real data.