A pressing concern faced by cancer patients is their prognosis under different treatment options. Considering a binary-treatment, e.g., to receive radiotherapy or not, the problem can be characterized as estimating the treatment effect of radiotherapy on the survival outcome of the patients. Estimating treatment effect from observational studies is a fundamental problem, yet it is still especially challenging due to the counterfactual and confounding problems. In this work, we show the importance of differentiating confounding factors from factors that only affect the treatment or the outcome, and propose a data-driven approach to learn and disentangle the latent factors into three disjoint sets for a more accurate estimating treatment estimator. Empirical validations on semi-synthetic benchmark and real-world datasets demonstrate the effectiveness of the proposed method.