In this paper, we utilize a downlink hybrid Non-Orthogonal Multiple Access (NOMA) framework to support multiple semantic and bit users within the communication network. The hybrid NOMA setup exploits both NOMA and Orthogonal Multiple Access (OMA) which has the benefit of enhancing Spectral Efficiency (SE) by allowing users to dynamically access the resources in multiple heterogeneous slots. This enables integrating semantic and bit users based on their channel gains, while adopting bit-to-semantic decoding order in slots including heterogeneous users. An optimization problem for the power allocation is formulated with the aim of maximizing the equivalent ergodic semantic SE with a constraint on the total available power of the Access Point (AP). The proposed algorithm uses NOMA in shared slots and OMA in bit-user-only slots. Simulation results validate the benefits of heterogeneous users hybrid NOMA setup in comparison to OMA-only for heterogeneous users.