Optimal resource allocation is a fundamental challenge for dense and heterogeneous wireless networks with massive wireless connections. Because of the non-convex nature of the optimization problem, it is computationally demanding to obtain the optimal resource allocation. Recently, deep reinforcement learning (DRL) has emerged as a promising technique in solving non-convex optimization problems. Unlike deep learning (DL), DRL does not require any optimal/ near-optimal training dataset which is either unavailable or computationally expensive in generating synthetic data. In this paper, we propose a novel centralized DRL based downlink power allocation scheme for a multi-cell system intending to maximize the total network throughput. Specifically, we apply a deep Q-learning (DQL) approach to achieve near-optimal power allocation policy. For benchmarking the proposed approach, we use a Genetic Algorithm (GA) to obtain near-optimal power allocation solution. Simulation results show that the proposed DRL-based power allocation scheme performs better compared to the conventional power allocation schemes in a multi-cell scenario.