Distribution-Aware GMD Transceiver Design for Probabilistic Shaping in MIMO

Multiple-input multiple-output (MIMO) transceiver design and probabilistic shaping (PS) are key enablers for high spectral efficiency in 6G wireless networks. This work proposes a distribution-aware MIMO transceiver optimized for PS constellation symbols, including a Bayesian geometric-mean decomposition (BGMD) precoder and a maximum a posteriori-VBLAST (MAP-VBLAST) detector. BGMD precoder incorporates PS priors into the derivation and equalizes layer gains to facilitate a single modulation and coding scheme for low-complexity transmissions while preserving channel capacity. MAP-VBLAST leverages these PS priors for optimal MAP detection within a successive interference cancellation (SIC) framework. Furthermore, a new codeword-to-layer mapping scheme, termed layer-contained MIMO (LC-MIMO), is proposed. By containing each codeblock (CB) within a single layer, LC-MIMO enables SIC at CB level, allowing the receiver to exploit the error-correction capability of channel coding to mitigate error propagation. Numerical results show that the BGMD transceiver with LC-MIMO achieves notable performance gains over state-of-the-art methods.