Gaussian Mixture Model Based Bayesian Learning for Sparse Channel Estimation in Orthogonal Time Frequency Space Modulated Systems

A novel Gaussian mixture model (GMM) aided sparse Bayesian learning (SBL) framework is proposed for channel state information (CSI) estimation in orthogonal time-frequency space (OTFS) modulated systems. The key attribute of the proposed algorithm lies in casting CSI recovery as an SBL inference problem, where posterior distributions are iteratively refined under a hierarchical GMM prior. Using this approach, the sparsity-inducing variances beneficially promote sparsity in the delay Doppler (DD) domain, while additionally augmenting the capability of SBL to exploit channel statistics more effectively. Moreover, to fully exploit the GMMs ability to approximate arbitrary probability density functions and model complex multipath fading scenarios, the channel statistics are represented using a complex Gaussian mixture. Simultaneously, the method leverages time-domain (TD) pilots without requiring wasteful DD domain guard intervals, thereby ensuring low pilot overhead and high spectral efficiency. The CSI recovered is subsequently applied in a linear minimum mean square error (MMSE) detector for reliable data detection. To benchmark performance, the Oracle-MMSE and the Bayesian Cramèr Rao lower bound (BCRLB) are also derived. Our simulation results demonstrate significant performance improvement over the state of the art sparse estimation methods.