We revisit one of the most basic and widely applicable techniques in the literature of differential privacy - the sparse vector technique [Dwork et al., STOC 2009]. Loosely speaking, this technique allows us to privately test whether the value of a given query is close to what we expect it would be (w.r.t. the input database), where we are allowed to test an unbounded number of queries as long as their value is indeed close to what we expected. After the first time in which this is not the case, the process halts. We present a modification to the sparse vector technique that allows for a more fine-tuned privacy analysis. As a result, in some cases we are able to continue with the process of testing queries even after the first time in which the value of the query did not meet our expectations. We demonstrate our technique by applying it to the shifting-heavy-hitters problem: On every time step, each of n users gets a new input, and the task is to privately identify all the current heavy-hitters. That is, on time step i, the goal is to identify all data elements x such that many of the users have x as their current input. We present an algorithm for this problem with improved error guarantees over what can be obtained using existing techniques. Specifically, the error of our algorithm depends on the maximal number of times that a singe user holds a heavy-hitter as input, rather than the total number of times in which a heavy-hitter exists.