Collective motion in animal groups, such as swarms of insects, flocks of birds, and schools of fish, are some of the most visually striking examples of emergent behavior. Empirical analysis of these behaviors in experiment or computational simulation primarily involves the use of "swarm-averaged" metrics or order parameters such as velocity alignment and angular momentum. Recently, tools from computational topology have been applied to the analysis of swarms to further understand and automate the detection of fundamentally different swarm structures evolving in space and time. Here, we show how the field of graph signal processing can be used to fuse these two approaches by collectively analyzing swarm properties using graph Fourier harmonics that respect the topological structure of the swarm. This graph Fourier analysis reveals hidden structure in a number of common swarming states and forms the basis of a flexible analysis framework for collective motion.