Constellation Design for Robust Interference Mitigation

This paper investigates symbol detection for single-carrier communication systems operating in the presence of additive interference with Nakagami-m statistics. Such interference departs from the assumptions underlying conventional detection methods based on Gaussian noise models and leads to detection mismatch that fundamentally affects symbol-level performance. In particular, the presence of random interference amplitude and non-uniform phase alters the structure of the optimal decision regions and renders standard Euclidean distance-based detectors suboptimal. To address this challenge, we develop the maximum-likelihood Gaussian-phase approximate (ML-G) detector, a low-complexity detection rule that accurately approximates maximum-likelihood detection while remaining suitable for practical implementation. The proposed detector explicitly incorporates the statistical properties of the interference and induces decision regions that differ significantly from those arising under conventional metrics. Building on the ML-G framework, we further investigate constellation design under interference-aware detection and formulate an optimization problem that seeks symbol placements that minimize the symbol error probability subject to an average energy constraint. The resulting constellations are obtained numerically and adapt to the interference environment, exhibiting non-standard and asymmetric structures as interference strength increases. Simulation results demonstrate clear symbol error probability gains over established benchmark schemes across a range of interference conditions, particularly in scenarios with dominant additive interference.