Autonomous excavation for hard or compact materials, especially irregular rigid objects, is challenging due to high variance of geometric and physical properties of objects, and large resistive force during excavation. In this paper, we propose a novel learning-based excavation planning method for rigid objects in clutter. Our method consists of a convolutional neural network to predict the excavation success and a sampling-based optimization method for planning high-quality excavation trajectories leveraging the learned prediction model. To reduce the sim2real gap for excavation learning, we propose a voxel-based representation of the excavation scene. We perform excavation experiments in both simulation and real world to evaluate the learning-based excavation planners. We further compare with two heuristic baseline excavation planners and one data-driven scene-independent planner. The experimental results show that our method can plan high-quality excavations for rigid objects in clutter and outperforms the baseline methods by large margins. As far as we know, our work presents the first learning-based excavation planner for cluttered and irregular rigid objects.