In this paper we propose a new model for detecting visual relationships. Our main technical novelty is a Box Attention mechanism that allows modelling pairwise interactions between objects in visual scenes using standard object detection pipelines. The resulting model is conceptually clean, expressive and relies on well-justified training and prediction procedures. Moreover, unlike previously proposed approaches, our model does not introduce any additional complex components or hyperparameters on top of those already required by the underlying detection model. We conduct an experimental evaluation on two challenging datasets, V-COCO and Visual Relationships, demonstrating strong quantitative and qualitative results.