Recommender system has intrinsic problems such as sparsity and fairness. Although it has been widely adopted for the past decades, research on fairness of recommendation algorithms has been largely neglected until recently. One important paradigm for resolving the issue is regularization. However, researchers have not been able to come up with a consensusly agreed regularization term like regularization framework in other fields such as Lasso or Ridge Regression. In this paper, we borrow concepts from information geometry and propose a new regularization-based fair algorithm called KL-Mat. The algorithmic technique leads to a more robust performance in accuracy performance such as MAE. More importantly, the algorithm produces much fairer results than vanilla matrix factorization approach. KL-Mat is fast, easy-to-implement and explainable.