Tertiary-Mode STAR-RIS for Secure NOMA: Integrating Transmission, Reflection, and Jamming

This paper investigates the physical layer security of a non-orthogonal multiple access (NOMA) system assisted by a tertiary-mode simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS), which can perform transmission, reflection, and jamming simultaneously. The system comprises a base station (BS) serving two users located on opposite sides of the STAR-RIS, assuming perfect channel state information (CSI) at the transmitter. To enhance secrecy performance, a subset of STAR-RIS elements is adaptively configured for jamming. A penalty-based alternating optimization algorithm is developed to jointly optimize the BS's active beamforming and the STAR-RIS's passive beamforming and mode selection. Simulation results demonstrate that the proposed design substantially improves the achievable sum rate and secrecy performance compared to conventional RIS-assisted and no-RIS benchmarks, highlighting the potential of tertiary-mode STAR-RIS for secure and efficient next-generation wireless communications.