Result ranking often affects customer satisfaction as well as the amount of exposure each product receives in recommendation systems (RecSys). Myopically maximizing customer satisfaction by ranking products only according to relevance will lead to unfair distribution of exposure for products, followed by unfair opportunities and economic gains for product producers. This unfairness will force producers to leave the system, and discourage new producers from coming in. Eventually, fewer purchase options would be left for customers and the overall transaction rates on e-commerce platforms would decrease. Thus, how to maintain a balance between ranking relevance and fairness is important to both producers and customers. In this paper, we focus on the task of exposure fairness in offline recommendation settings. We demonstrate that existing methods for amortized fairness optimization are suboptimal for offline recommendation because they fail to utilize the prior knowledge of customers. We further propose a novel fair recommendation algorithm to reach a better balance between exposure fairness and recommendation performance. Extensive experiments on three real-world datasets demonstrate that our method significantly outperforms the state-of-the-art fair ranking algorithm in terms of fairness-performance trade off from both individual level and group level.