Autonomous aerial swarm remains a grand challenge in robotics. Existing works in this field can be categorized as centralized and decentralized. Centralized methods suffer from scale dilemmas, while decentralized ones often lead to poor planning quality. In this paper, we propose an enhanced decentralized autonomous aerial swarm system with group planning. According to the spatial distribution of agents, the system dynamically divides the swarm into several groups and isolated agents. For conflicts within each group, we propose a novel coordination mechanism named group planning. The group planning consists of efficient multi-agent pathfinding and trajectory joint optimization, which can significantly improve the planning quality and success rate. We demonstrate simulations and real-world experiments that our method not only has applicability for a large-scale swarm, but also has top-level planning quality.