Classical approaches in learning theory are often seen to yield very loose generalization bounds for deep neural networks. Using the example of "stability and generalization" \citep{bousquet2002stability}, however, we demonstrate that generalization bounds can be significantly improved by taking into account refined characteristics of modern neural networks. Specifically, this paper proposes a new notion of algorithmic stability termed \textit{locally elastic stability} in light of a certain phenomenon in the training of neural networks \citep{he2020local}. We prove that locally elastic stability implies a tighter generalization bound than that derived based on uniform stability in many situations. When applied to deep neural networks, our new generalization bound attaches much more meaningful confidence statements to the performance on unseen data than existing algorithmic stability notions, thereby shedding light on the effectiveness of modern neural networks in real-world applications.