Secure and Robust Beamforming Design for STAR-RIS-aided MU-MIMO ISAC Systems

Simultaneous transmitting and reflecting reconfigurable intelligent surfaces (STAR-RIS) offer a transformative approach for integrated sensing and communication (ISAC) systems, particularly for enhancing physical layer security (PLS). This paper investigates a robust, secure downlink transmission framework for a STAR-RIS empowered multi-user (MU) multiple-input multiple-output (MU-MIMO) system, where a multi-antenna dual-function radar and communication base station (DFRC-BS) that simultaneously transmits confidential messages to multiple intended users (IUs) and performs target sensing in the presence of malicious eavesdroppers. To optimize system security, we formulate a worst-case robust beamforming problem to maximize the secrecy rate. This formulation jointly designs the active transmit beamforming at the BS and the passive reflection, transmission coefficients at the STAR-RIS, adheres to transmit power budgets, user quality-of-service (QoS) thresholds, sensing signal-to-interference-plus-noise ratio (SINR) requirements, maximum tolerable eavesdropping leakage, and practical phase shifts constraints. To efficiently tackle the formulated problem, we develop an alternating optimization (AO) algorithm. Specifically, the S-procedure is employed to solve semi-infinite channel uncertainty constraints, while semidefinite relaxation (SDR) and penalty convex-concave programming (CCP) are applied to obtain tractable suboptimal solutions. Extensive simulation results validate the efficacy of the proposed framework and demonstrate significant improvement in spectral efficiency compared to conventional reflecting-only RIS (R-RIS) systems under stringent sensing conditions.