In cities worldwide, cars cause health and traffic problems which could be partly mitigated through an increased modal share of bicycles. Many people, however, avoid cycling due to a lack of perceived safety. For city planners, addressing this is hard as they lack insights into where cyclists feel safe and where they do not. To gain such insights, we have in previous work proposed the crowdsourcing platform SimRa, which allows cyclists to record their rides and report near miss incidents via a smartphone app. In this paper, we present CycleSense, a combination of signal processing and Machine Learning techniques, which partially automates the detection of near miss incidents. Using the SimRa data set, we evaluate CycleSense by comparing it to a baseline method used by SimRa and show that it significantly improves incident detection.