Collective motion among biological organisms such as insects, fish, and birds has motivated considerable interest not only in biology but also in distributed robotic systems. In a robotic or biological swarm, anomalous agents (whether malfunctioning or nefarious) behave differently than the normal agents and attempt to hide in the "chaos" of the swarm. By defining a graph structure between agents in a swarm, we can treat the agents' properties as a graph signal and use tools from the field of graph signal processing to understand local and global swarm properties. Here, we leverage this idea to show that anomalous agents can be effectively detected using their impacts on the graph Fourier structure of the swarm.