Surgical phase recognition is a fundamental task in computer-assisted surgery systems. Most existing works require expensive frame-wise annotations, which is very time-consuming. In this paper, we introduce timestamp supervision to surgical phase recognition for the first time, which only requires randomly labeling one frame for each phase in a video. With timestamp supervision, current methods in natural videos aim to generate pseudo labels of full frames. However, due to the surgical videos containing ambiguous boundaries, these methods would generate many noisy and inconsistent pseudo labels, leading to limited performance. We argue that less is more in surgical phase recognition,~\ie, less but discriminative pseudo labels outperform full but ambiguous frames. To this end, we propose a novel method called uncertainty-aware temporal diffusion to generate trustworthy pseudo labels. Our approach evaluates the confidence of generated pseudo labels based on uncertainty estimation. Then, we treat the annotated frames as anchors and make pseudo labels diffuse to both sides, starting from anchors and stopping at the high-uncertainty frames. In this way, our proposed method can generate contiguous confident pseudo labels while discarding the uncertain ones. Extensive experiments demonstrate that our method not only significantly save annotation cost, but also outperforms fully supervised methods. Moreover, our proposed approach can be used to clean noisy labels near boundaries and improve the performance of the current surgical phase recognition methods.