J. B. Fourier in his \emph{Théorie Analytique de la Chaleur} of 1822 introduced, amongst other things, two ideas that have made a fundamental impact in fields as diverse as Mathematical Physics, Electrical Engineering, Computer Science, and Music. The first one of these, a method to find the coefficients for a trigonometric series describing an arbitrary function, was very early on picked up by G. Ohm and H. Helmholtz as the foundation for a theory of \emph{musical tones}. The second one, which is described by Fourier's double integral, became the basis for treating certain kinds of infinity in discontinuous functions, as shown by A. De Morgan in his 1842 \emph{The Differential and Integral Calculus}. Both make up the fundamental basis for what is now commonly known as the \emph{Fourier theorem}. With the help of P. A. M. Dirac's insights into the nature of these infinities, we can have a compact description of the frequency spectrum of a function of time, or conversely of a waveform corresponding to a given function of frequency. This paper, using solely primary sources, takes us from the physics of heat propagation to the modern theory of musical signals. It concludes with some considerations on the inherent duality of time and frequency emerging from Fourier's theorem.