Radar sensor provides lighting and weather invariant sensing, which is naturally suitable for long-term localization in outdoor scenes. On the other hand, the most popular available map currently is built by lidar. In this paper, we propose a deep neural network for end-to-end learning of radar localization on lidar map to bridge the gap. We first embed both sensor modals into a common feature space by a neural network. Then multiple offsets are added to the map modal for similarity evaluation against the current radar modal, yielding the regression of the current pose. Finally, we apply this differentiable measurement model to a Kalman filter to learn the whole sequential localization process in an end-to-end manner. To validate the feasibility and effectiveness, we employ multi-session multi-scene datasets collected from the real world, and the results demonstrate that our proposed system achieves superior performance over 90km driving, even in generalization scenarios where the model training is in UK, while testing in South Korea. We also release the source code publicly.