Intelligent reflecting surfaces (IRSs) are being studied as possible low-cost energy-efficient alternatives to active relays, with the goal of solving the coverage issues of millimeter wave (mmWave) and terahertz (THz) network deployments. In the literature, these surfaces are often studied by idealizing their characteristics. Notably, it is often assumed that IRSs can tune with arbitrary frequency the phase-shifts induced by their elements, thanks to a wire-like control channel to the next generation node base (gNB). Instead, in this work we investigate an IRS-aided time division multiple access (TDMA) cellular network, where the reconfiguration of the IRS may entail an energy or communication cost, and we aim at limiting the number of reconfigurations over time. We develop a clustering-based heuristic scheduling, which optimizes the system sum-rate subject to a given number of reconfigurations within the TDMA frame. To such end, we first cluster user equipments (UEs) with a similar optimal IRS configuration. Then, we compute an overall IRS cluster configuration, which can be thus kept constant while scheduling the whole UEs cluster. Numerical results show that our approach is effective in supporting IRSs-aided systems with practical constraints, achieving up to 85% of the throughput obtained by an ideal deployment, while providing a 50% reduction in the number of IRS reconfigurations.