Reconfigurable Intelligent Surfaces (RISs) are one of the key emerging 6th Generation (6G) technologies that are expected to improve the link budgets between transmitters and receivers by adding artificial propagation paths. In such re-configured propagation environment, Downlink (DL) Multi-User Multi-Input Multi-Output (MU-MIMO) brings capacity improvement to cellular networks. It benefits from the spatial dimension offered by MIMO systems to enable simultaneous transmission of independent data streams to multiple users on the same radio resources by applying appropriate Beamforming (BF) schemes. However, in some cases, serving the same subset of users for a long period of time may cause some undesired regions where the average Electromagnetic Field Exposure (EMFE) exceeds the regulatory limits. To address this challenge, we propose in this paper a novel Electromagnetic Field (EMF) aware MU-MIMO BF scheme that aims to optimize the overall capacity under EMF constraints in RIS-aided cellular networks.