In cellular networks, resource allocation is performed in a centralized way, which brings huge computation complexity to the base station (BS) and high transmission overhead. This paper investigates the distributed resource allocation scheme for cellular networks to maximize the energy efficiency of the system in the uplink transmission, while guaranteeing the quality of service (QoS) for cellular users. Particularly, to cope the fast varying channels in wireless communication environment, we propose a robust federated reinforcement learning (FRL_suc) framework to enable local users to perform distributed resource allocation in items of transmit power and channel assignment by the guidance of the local neural network trained at each user. Analysis and numerical results show that the proposed FRL_suc framework can lower the transmission overhead and offload the computation from the central server to the local users, while outperforming the conventional multi-agent reinforcement learning algorithm in terms of EE, and is more robust to channel variations.