From our experiences in the past, we have seen that the growth of cities is very much dependent on the transportation networks. In mega cities, transportation networks determine to a significant extent as to where the people will move and houses will be built. Hence, transportation network data is crucial to an urban growth prediction system. Existing works have used manually derived distance based features based on the road networks to build models on urban growth. But due to the non-generic and laborious nature of the manual feature engineering process, we can shift to End-to-End systems which do not rely on manual feature engineering. In this paper, we propose a method to integrate road network data to an existing Rule based End-to-End framework without manual feature engineering. Our method employs recurrent neural networks to represent road networks in a structured way such that it can be plugged into the previously proposed End-to-End framework. The proposed approach enhances the performance in terms of Figure of Merit, Producer's accuracy, User's accuracy and Overall accuracy of the existing Rule based End-to-End framework.