Location information is expected to be the key to meeting the needs of communication and context-aware services in 6G systems. User localization is achieved based on delay and/or angle estimation using uplink or downlink pilot signals. However, hardware impairments (HWIs) distort the signals at both the transmitter and receiver sides and thus affect the localization performance. While this impact can be ignored at lower frequencies where HWIs are less severe, modeling and analysis efforts are needed for 6G to evaluate the localization degradation due to HWIs. In this work, we model various types of impairments and conduct a misspecified Cram\'er-Rao bound analysis to evaluate the HWI-induced performance loss. Simulation results with different types of HWIs show that each HWI leads to a different level of degradation in angle and delay estimation performance.