Map matching is a key part of many GIS applications, linking observed GPS traces to road networks via a map. But when that map contains errors such as missing or mislabeled roads, map matching can give poor or even misleading results. Here, an approach to tracking vehicles able to move both on and off known road networks is introduced that efficiently unifies existing hidden Markov model (HMM) approaches for map matching and standard free-space tracking methods (e.g. Kalman smoothing) in a principled way. In addition to avoiding generating misleading map-matching output, this approach has applications in learning map information from GPS traces, for example detecting unmapped or incorrectly mapped roads and parking lots. The approach is a form of interacting multiple model (IMM) filter subject to an additional assumption on the type of model interaction permitted. This allows an efficient formulation, here termed a semi-interacting multiple model (sIMM) filter. A forward filter (suitable for realtime tracking) and backward MAP sampling step (suitable for MAP trajectory inference and map matching) are described. The framework set out here is agnostic to the specific tracking models used, and makes clear how to replace these components with others of a similar type.