For driverless train operation on mainline railways, several tasks need to be implemented by technical systems. One of the most challenging tasks is to monitor the train's driveway and its surroundings for potential obstacles due to long braking distances. Machine learning algorithms can be used to analyze data from vision sensors such as infrared (IR) and visual (RGB) cameras, lidars, and radars to detect objects. Such algorithms require large amounts of annotated data from objects in the rail environment that may pose potential obstacles, as well as rail-specific objects such as tracks or catenary poles, as training data. However, only very few datasets are publicly available and these available datasets typically involve only a limited number of sensors. Datasets and trained models from other domains, such as automotive, are useful but insufficient for object detection in the railway context. Therefore, this publication presents OSDaR23, a multi-sensor dataset of 21 sequences captured in Hamburg, Germany, in September 2021. The sensor setup consisted of multiple calibrated and synchronized IR/RGB cameras, lidars, a radar, and position and acceleration sensors front-mounted on a railway vehicle. In addition to raw data, the dataset contains 204091 polyline, polygonal, rectangle and cuboid annotations for 20 different object classes. This dataset can also be used for tasks going beyond collision prediction, which are listed in this paper.