Semantic segmentation is paramount to accomplish many scene understanding applications such as autonomous driving. Although deep convolutional networks have already achieved encouraging results in semantic segmentation compared to traditional methods, there is still large room for further improvement. In this paper, we propose a preferred solution, which incorporates Luminance, Depth and color information by a Fusion-based network named LDFNet. It includes a distinctive encoder sub-network to process the depth maps and further employs the luminance images to assist the depth information in a process. LDFNet achieves very competitive results compared to the other state-of-art systems on the challenging Cityscapes dataset, while it maintains an inference speed faster than most of the existing top-performing networks. The experimental results show the effectiveness of the proposed information-fused approach and the potential of LDFNet for road scene understanding tasks.