This paper investigates the optimization of reconfigurable intelligent surface (RIS) in an integrated sensing and communication (ISAC) system. \red{To meet the demand of growing number of devices, power domain non-orthogonal multiple access (NOMA) is considered. However, traditional NOMA with a large number of devices is challenging due to large decoding delay and propagation error introduced by successive interference cancellation (SIC). Thus, OMA is integrated into NOMA to support more devices}. We formulate a max-min problem to optimize the sensing beampattern \red{with constraints on communication rate}, through joint power allocation, active beamforming and RIS phase shift design. To solve the non-convex problem with a non-smooth objective function, we propose a low complexity alternating optimization (AO) algorithm, where a closed form expression for the intra-cluster power allocation (intra-CPA) is derived, and penalty and successive convex approximation (SCA) methods are used to optimize the beamforming and phase shift design. Simulation results show the effectiveness of the proposed algorithm in terms of improving minimum beampattern gain (MBPG) compared with other baselines. Furthermore, the trade-off between sensing and communication is analyzed and demonstrated in the simulation results.