Pedestrian detection in a crowd is very challenging due to vastly different scales and poor conditions. Pedestrian detectors are generally designed by extending generic object detectors, where Non-maximum suppression (NMS) is a standard but critical post-processing step for refining detection results. In this paper, we propose CSID: a Center, Scale, Identity-and-Density-aware pedestrian detector with a novel Identity-and-Density-aware NMS (ID-NMS) algorithm to refine the results of anchor-free pedestrian detection. Our main contributions in this work include (i) a novel Identity and Density Map (ID-Map) which converts each positive instance into a feature vector to encode both identity and density information simultaneously, (ii) a modified optimization target in defining ID-loss and addressing the extremely class imbalance issue during training, and (iii) a novel ID-NMS algorithm by considering both identity and density information of each predicted box provided by ID-Map to effectively refine the detection results. We evaluate the proposed CSID pedestrian detector using the novel ID-NMS technique and achieve new state-of-the-art results on two benchmark data sets (CityPersons and CrowdHuman) for pedestrian detection.