Autonomous vehicles must be capable of handling the occlusion of the environment to ensure safe and efficient driving. In urban environment, occlusion often arises due to other vehicles obscuring the perception of the ego vehicle. Since the occlusion condition can impact the trajectories of vehicles, the behavior of other vehicles is helpful in making inferences about the occlusion as a remedy for perceptual deficiencies. This paper introduces a novel social occlusion inference approach that learns a mapping from agent trajectories and scene context to an occupancy grid map (OGM) representing the view of ego vehicle. Specially, vectorized features are encoded through the polyline encoder to aggregate features of vectors into features of polylines. A transformer module is then utilized to model the high-order interactions of polylines. Importantly, occlusion queries are proposed to fuse polyline features and generate the OGM without the input of visual modality. To verify the performance of vectorized representation, we design a baseline based on a fully transformer encoder-decoder architecture mapping the OGM with occlusion and historical trajectories information to the ground truth OGM. We evaluate our approach on an unsignalized intersection in the INTERACTION dataset, which outperforms the state-of-the-art results.