Semantic segmentation has achieved significant progress but is still challenging due to the complex scene, object occlusion, and so on. Some research works have attempted to use extra information such as depth information to help RGB based semantic segmentation. However, extra information is usually unavailable for the test images. Inspired by learning using privileged information, in this paper, we only leverage the depth information of training images as privileged information in the training stage. Specifically, we rely on depth information to identify the hard pixels which are difficult to classify, by using our proposed Depth Prediction Error (DPE) and Depth-dependent Segmentation Error (DSE). By paying more attention to the identified hard pixels, our approach achieves the state-of-the-art results on two benchmark datasets and even outperforms the methods which use depth information of test images.