The focus of this paper is the estimation of a delay between two signals. Such a problem is common in signal processing and particularly challenging when the delay is non-stationary in nature. Our proposed solution is based on an all-pass filter framework comprising of two elements: a time delay is equivalent to all-pass filtering and an all-pass filter can be represented in terms of a ratio of a finite impulse response (FIR) filter and its time reversal. Using these elements, we propose an adaptive filtering algorithm with an LMS style update that estimates the FIR filter coefficients and the time delay. Specifically, at each time step, the algorithm updates the filter coefficients based on a gradient descent update and then extracts an estimate of the time delay from the filter. We validate our algorithm on synthetic data demonstrating that it is both accurate and capable of tracking time-varying delays.