Utilizing Improper Gaussian Signaling for Downlink Rate-Splitting Multiple Access with Imperfect Successive Interference Cancellation

To mitigate the residual interference from imperfect successive interference cancellation (SIC) in Rate-Splitting Multiple Access (RSMA), this paper incorporates improper Gaussian signaling (IGS) into the downlink RSMA framework. Unlike existing RSMA--IGS works that embed impropriety within IQ-imbalanced frameworks, we show that IGS alone effectively counters SIC-induced residual interference. For a basic SISO setup with IGS on the common stream and PGS on private streams, we establish three key results: the optimal impropriety degree for private rate maximization attains its maximum; closed-form optimal solutions with rigorous monotonicity conditions are derived for common rate maximization; and a soft actor-critic (SAC) algorithm is developed for the non-convex sum rate problem. Numerical results show that IGS consistently outperforms PGS, with the gain widening as SIC imperfection increases.