Abstract:Next-generation wireless communications promise transformative technologies such as massive multiple-input multiple-output (MIMO), reconfigurable intelligent surfaces (RIS), integrated sensing and communication (ISAC), and fluid antenna systems (FAS). However, deploying these technologies is hindered by large-scale optimization problems with nonconvex constraints. Conventional Euclidean-space methods rely on approximations or relaxations, which degrade performance and incur substantial computational costs. Riemannian manifold optimization (RMO) offers a powerful alternative that directly operates on the manifold defined by the geometric constraints. This approach inherently satisfies the constraints at every optimization step, thereby avoiding the performance degradation and substantial computational costs. In this paper, we first elaborate on the principles of RMO, including the fundamental concepts, tools, and methods, emphasizing its effectiveness for nonconvex problems. We then introduce its applications in advanced wireless communications, showing how constrained problems are reformulated on their natural manifolds and solved using tailored RMO algorithms. Furthermore, we present a case study on secure beamforming in an FAS-assisted non-orthogonal multiple access (NOMA) system, demonstrating RMO's superiority over conventional methods in terms of both performance and computational efficiency.