Relative localization is a prerequisite for the cooperation of aerial swarms. The vision-based approach has been investigated owing to its scalability and independence on communication. However, the limited field of view (FOV) inherently restricts the performance of vision-based relative localization. Inspired by bird flocks in nature, this letter proposes a novel distributed active vision-based relative localization framework for formation control in aerial swarms. Aiming at improving observation quality and formation accuracy, we devise graph-based attention planning (GAP) to determine the active observation scheme in the swarm. Then active detection results are fused with onboard measurements from UWB and VIO to obtain real-time relative positions, which further improve the formation control performance. Real-world experiments show that the proposed active vision system enables the swarm agents to achieve agile flocking movements with an acceleration of 4 $m/s^2$ in circular formation tasks. A 45.3 % improvement of formation accuracy has been achieved compared with the fixed vision system.