Multilayer Non-Terrestrial Networks with Spectrum Access aided by Beyond-Diagonal RIS

In this work, we study a multi-user NTN in which a satellite serves as the primary network and a high-altitude platform station (HAPS) operates as the secondary network, acting as a cognitive radio. To reduce the cost, complexity, and power consumption of conventional antenna arrays, we equip the HAPS with a transmissive BD-RIS antenna front end. We then formulate a joint optimization problem for the BD-RIS phase response and the HAPS transmit power allocation under strict per-user interference temperature constraints. To tackle the resulting highly nonconvex problem, we propose an alternating-optimization framework: the power-allocation subproblem admits a closed-form, water-filling-type solution derived from the Karush-Kuhn-Tucker (KKT) conditions, while the BD-RIS configuration is refined via Riemannian manifold optimization. Simulation results show significant gains in data rate and interference suppression over diagonal RIS-assisted benchmarks, establishing BD-RIS as a promising enabler for future multilayer NTNs.