Recent advancements in quantum computing have shown promising computational advantages in many problem areas. As one of those areas with increasing attention, hybrid quantum-classical machine learning systems have demonstrated the capability to solve various data-driven learning tasks. Recent works show that parameterized quantum circuits (PQCs) can be used to solve challenging reinforcement learning (RL) tasks with provable learning advantages. While existing works yield potentials of PQC-based methods, the design choices of PQC architectures and their influences on the learning tasks are generally underexplored. In this work, we introduce EQAS-PQC, an evolutionary quantum architecture search framework for PQC-based models, which uses a population-based genetic algorithm to evolve PQC architectures by exploring the search space of quantum operations. Experimental results show that our method can significantly improve the performance of hybrid quantum-classical models in solving benchmark reinforcement problems. We also model the probability distributions of quantum operations in top-performing architectures to identify essential design choices that are critical to the performance.