This paper presents Polylidar, an efficient algorithm to extract non-convex polygons from 2D point sets. Polylidar is able to extract multiple disjoint polygons and capture interior holes. The algorithm begins by triangulating the point set and filtering triangles by user configurable parameters such as triangle edge length. Next, connected triangles are extracted into triangular meshes representing the shape of the point set. The key to Polylidar's speed, and the main contribution of this paper, is in efficiently transforming each triangular mesh into a concave polygon. This paper describes Polylidar and provides benchmarks to comparatively evaluate its speed and accuracy. Results show good accuracy and $\sim$ 4 times speedup compared to other concave polygon extraction methods.