Symbol-Level Mask-Compliant Hybrid Precoding for Multi-User MIMO-OFDM Systems

Millimeter-wave (mmWave) technology is a crucial enabler for next-generation networks because it offers substantially greater available bandwidth. mmWave multiple-input multiple-output (MIMO) systems cannot rely solely on fully digital precoding due to hardware costs. As a result, hybrid precoding, which combines digital baseband processing with RF precoding, has emerged as a practical solution that balances performance and implementation complexity. As mmWave links typically operate over wideband, frequency-selective channels, orthogonal frequency-division multiplexing (OFDM) is commonly used to mitigate dispersive effects, yet OFDM introduces practical drawbacks, including out-of-band (OOB) emissions from abrupt spectral transitions among subcarriers and additional spectral leakage induced by windowing. Moreover, nonideal phase shifters (PS) in the RF transmit precoder and the user combiner impose inherent implementation limits that result in phase errors. We investigate robust joint digital--RF precoder design for minimizing the downlink sum mean-squared error (MSE) in hybrid multi-user (MU) MIMO--OFDM systems subject to maximum transmit-power, clipping, and OOB spectral-mask constraints. The resulting optimization is nonconvex and challenging to solve. To address this, we develop a minimum mean-squared error (MMSE) based block coordinate descent (BCD) algorithm that alternates between updating the transmitter-side digital--RF precoders and the user-side digital--RF combiners. For each BCD subproblem, we propose computationally efficient and scalable, closed-form solution strategies suitable for practical implementation. Extensive simulations validate the proposed methods and show clear performance improvements over established benchmark schemes.