Installing more base stations (BSs) into the existing cellular infrastructure is an essential way to provide greater network capacity and higher data rate in the 5th-generation cellular networks (5G). However, a non-negligible amount of population is concerned that such network densification will generate a notable increase in exposure to electric and magnetic fields (EMF) over the territory. In this paper, we analyze the downlink, uplink, and joint downlink&uplink exposure induced by the radiation from BSs and personal user equipment (UE), respectively, in terms of the received power density and exposure index. In our analysis, we consider the EMF restrictions set by the regulatory authorities such as the minimum distance between restricted areas (e.g., schools and hospitals) and BSs, and the maximum permitted exposure. Exploiting tools from stochastic geometry, mathematical expressions for the coverage probability and statistical EMF exposure are derived and validated. Tuning the system parameters such as the BS density and the minimum distance from a BS to restricted areas, we show a trade-off between reducing the population's exposure to EMF and enhancing the network coverage performance. Then, we formulate optimization problems to maximize the performance of the EMF-aware cellular network while ensuring that the EMF exposure complies with the standard regulation limits with high probability. For instance, the exposure from BSs is two orders of magnitude less than the maximum permissible level when the density of BSs is less than 20 BSs/km2.