We propose a variational inference (VI)-based channel state information (CSI) estimation approach in a fully-passive reconfigurable intelligent surface (RIS)-aided mmWave single-user single-input multiple-output (SIMO) communication system. Specifically, we first propose a VI-based joint channel estimation method to estimate the user-equipment (UE) to RIS (UE-RIS) and RIS to base station (RIS-BS) channels using uplink training signals in a passive RIS setup. However, updating the phase-shifts based on the instantaneous CSI (I-CSI) leads to a high signaling overhead especially due to the short coherence block of the UE-RIS channel. Therefore, to reduce the signaling complexity, we propose a VI-based method to estimate the RIS-BS channel along with the covariance matrix of the UE-RIS channel that remains quasi-static for a longer period than the instantaneous UE-RIS channel. In the VI framework, we approximate the posterior of the channel gains/covariance matrix with convenient distributions given the received uplink training signals. Then, the learned distributions, which are close to the true posterior distributions in terms of Kullback-Leibler divergence, are leveraged to obtain the maximum a posteriori (MAP) estimation of the considered CSI. The simulation results demonstrate that MAP channel estimation using approximated posteriors yields a capacity that is close to the one achieved with true posteriors, thus demonstrating the effectiveness of the proposed methods. Furthermore, our results show that estimating the channel covariance matrix improves the spectral efficiency by reducing the pilot signaling required to obtain the phase-shifts for the RIS elements in a channel-varying environment.