Current face recognition systems achieved high progress on several benchmark tests. Despite this progress, recent works showed that these systems are strongly biased against demographic sub-groups. Consequently, an easily integrable solution is needed to reduce the discriminatory effect of these biased systems. Previous work introduced fairness-enhancing solutions that strongly degrades the overall system performance. In this work, we propose a novel fair score normalization approach that is specifically designed to reduce the effect of bias in face recognition and subsequently lead to a significant overall performance boost. Our hypothesis is built on the notation of individual fairness by designing a normalization approach that leads to treating "similar" individuals "similarly". Experiments were conducted on two publicly available datasets captured under controlled and in-the-wild circumstances. The results show that our fair normalization approach enhances the overall performance by up to 14.8% under intermediate false match rate settings and up to 30.7% under high security settings. Our proposed approach significantly reduces the errors of all demographic groups, and thus reduce bias. Especially under in-the-wild conditions, we demonstrated that our fair normalization method improves the recognition performance of the effected population sub-groups by 31.6%. Unlike previous work, our proposed fairness-enhancing solution does not require demographic information about the individuals, leads to an overall performance boost, and can be easily integrated in existing biometric systems.