This paper presents a factor graph formulation and particle-based sum-product algorithm (SPA) for scalable detection and tracking of extended objects. The proposed method efficiently performs probabilistic multiple-measurement to object association, represents object extents by random matrices, and introduces the states of newly detected objects dynamically. Scalable detection and tracking of objects is enabled by modeling association uncertainty by measurement-oriented association variables and newly detected objects by a Poisson birth process. Contrary to conventional extended object tracking (EOT) methods with random-matrix models, a fully particle-based approach makes it possible to represent the object extent by different geometric shapes. The proposed method can reliably determine the existence and track a large number of closely-spaced extended objects without gating and clustering of measurements. We demonstrate significant performance advantages of our method compared to the recently proposed Poisson multi-Bernoulli mixture filter in a challenging tracking scenario with ten closely-spaced extended objects.