Off-topic spoken response detection, the task aiming at assessing whether the response is off-topic for the corresponding prompt, is important for automatic spoken assessment system. In many real-world educational applications, off-topic spoken response detection algorithm is required to achieve high recall not only on seen prompts but also on prompts that are unseen during the training process. In this paper, we propose a novel approach for off-topic spoken response detection with high off-topic recall on both seen and unseen prompts. We introduce a novel model, Gated Convolutional Bidirectional Attention-based Model (GCBi-AM), where bi-attention mechanism and convolutions are applied to extract topic words of prompt and key-phrases of a response, and gated unit and residual connections between each major layer are introduced to better represent the relevance of response and prompt. Moreover, a new negative sampling method is also proposed to augment training data. Experiment results demonstrate that our new approach can achieve significant improvements on detecting off-topic responses with extremely high on-topic recall rate, for both seen and unseen prompts.