We propose ProoFVer, a proof system for fact verification using natural logic. The textual entailment model in ProoFVer is a seq2seq model generating valid natural-logic based logical inferences as its proofs. The generation of proofs makes ProoFVer an explainable system. The proof consists of iterative lexical mutations of spans in the claim with spans in a set of retrieved evidence sentences. Further, each such mutation is marked with an entailment relation using natural logic operators. The veracity of a claim is determined solely based on the sequence of natural logic relations present in the proof. By design, this makes ProoFVer a faithful by construction system that generates faithful explanations. ProoFVer outperforms existing fact-verification models, with more than two percent absolute improvements in performance and robustness. In addition to its explanations being faithful, ProoFVer also scores high on rationale extraction, with a five point absolute improvement compared to attention-based rationales in existing models. Finally, we find that humans correctly simulate ProoFVer's decisions more often using the proofs, than the decisions of an existing model that directly use the retrieved evidence for decision making.