Localization (position and orientation estimation) is envisioned as a key enabler to satisfy the requirements of communication and context-aware services in the sixth generation (6G) communication systems. User localization can be achieved based on delay and angle estimation using uplink or downlink pilot signals. However, hardware impairments (HWIs) distort the signals at both the transmitter and receiver sides and thus affect the localization performance. While this impact can be ignored at lower frequencies where HWIs are less severe, and the localization requirements are not stringent, modeling and analysis efforts are needed for high-frequency 6G bands (e.g., sub-THz) to assess degradation in localization accuracy due to HWIs. In this work, we model various types of impairments for a sub-THz multiple-input-multiple-output communication system and conduct a misspecified Cram\'er-Rao bound analysis to evaluate HWI-induced performance losses in terms of angle/delay estimation and the resulting 3D position/orientation estimation error. Complementary to the localization analysis, we also investigate the effect of individual and overall HWIs on communication in terms of symbol error rate (SER). Our extensive simulation results demonstrate that each type of HWI leads to a different level of degradation in angle and delay estimation performance. The prominent factors on delay estimation (e.g., phase noise and carrier frequency offset) will have a dominant negative effect on SER, while the impairments affecting only the angle estimation (e.g., mutual coupling and antenna displacement) induce slight degradation in SER performance.