Robust and safe self-driving in complex and dynamic environments is quite challenging due to the requirement of scalable driving policies against the wide variety of traffic scenarios (e,g., road topologies, traffic rules and interaction with road agents). In this area, traditional modular frameworks scale poorly in new environments, and require tremendous and iterative hand-tuning of rules and parameters to maintain performance in all foreseeable scenarios. Recently, deep-learning based self-driving methods have shown promising results with better generalization capability but less hand engineering effort. However, most of the previous methods are trained and evaluated in limited and simple environments with scattered tasks, such as lane-following, autonomous braking and conditional driving. In this paper, we propose a graph-based deep network to achieve unified and scalable self-driving in diverse dynamic environments. The extensive evaluation results show that our model can safely navigate the vehicle in a large variety of urban, rural, and highway areas with dense traffic while obeying traffic rules. Specifically, more than 7,500 km of closed-loop driving evaluation is conducted in dynamic simulation environments, in which our method can handle complex driving situations, and achieve higher success rates (73.5%-83.2%) and driving scores than the baselines.