In this paper, the issue of model uncertainty in safety-critical control is addressed with a data-driven approach. For this purpose, we utilize a structure of an input-ouput linearization controller based on a nominal model, and Control Barrier Function (CBF) and Control Lyapunov Function (CLF)-based control methods. Specifically, a novel Reinforcement Learning framework which learns the model uncertainty in CBF and CLF constraints, as well as other dynamic constraints is proposed. The trained policy is combined with the nominal model-based CBF-CLF-QP, resulting in the Reinforcement Learning based CBF-CLF-QP (RL-CBF-CLF-QP), which now addresses the problem of uncertainty in the safety constraints. The performance of the proposed method is validated by testing it on an underactuated nonlinear bipedal robot walking on randomly spaced stepping stones with one step preview.