Face recognition technology has demonstrated tremendous progress over the past few years, primarily due to advances in representation learning. As we witness the widespread adoption of these systems, it is imperative to consider the security of face representations. In this paper, we explore the practicality of using a fully homomorphic encryption based framework to secure a database of face templates. This framework is designed to preserve the privacy of users and prevent information leakage from the templates, while maintaining their utility through template matching directly in the encrypted domain. Additionally, we also explore a batching and dimensionality reduction scheme to trade-off face matching accuracy and computational complexity. Experiments on benchmark face datasets (LFW, IJB-A, IJB-B, CASIA) indicate that secure face matching can be practically feasible (16 KB template size and 0.01 sec per match pair for 512-dimensional features from SphereFace) while exhibiting minimal loss in matching performance.