Deep learning algorithms are preferable for rectal tumor segmentation. However, it is still a challenge task to accurately segment and identify the locations and sizes of rectal tumors by using deep learning methods. To increase the capability of extracting enough feature information for rectal tumor segmentation, we propose a Covariance Self-Attention Dual Path UNet (CSA-DPUNet). The proposed network mainly includes two improvements on UNet: 1) modify UNet that has only one path structure to consist of two contracting path and two expansive paths (nam new network as DPUNet), which can help extract more feature information from CT images; 2) employ the criss-cross self-attention module into DPUNet, meanwhile, replace the original calculation method of correlation operation with covariance operation, which can further enhances the characterization ability of DPUNet and improves the segmentation accuracy of rectal tumors. Experiments illustrate that compared with the current state-of-the-art results, CSA-DPUNet brings 15.31%, 7.2%, 11.8%, and 9.5% improvement in Dice coefficient, P, R, F1, respectively, which demonstrates that our proposed CSA-DPUNet is effective for rectal tumor segmentation.