This work focuses on the use of reconfigurable intelligent surface (RIS) in dual-function radar-communication (DFRC) systems to improve communication capacity and sensing precision, and enhance coverage for both functions. In contrast to most of the existing RIS aided DFRC works where the RIS is modeled as a diagonal phase shift matrix and can only reflect signals to half space, we propose a novel beyond diagonal RIS (BD-RIS) aided DFRC system. Specifically, the proposed BD-RIS supports the hybrid reflecting and transmitting mode, and is compatible with flexible single/group/fully-connected architectures, enabling the system to realize full-space coverage. To achieve the expected benefits, we jointly optimize the transmit waveform, the BD-RIS coefficients, and sensing receive filters, by maximizing the minimum signal-to-clutter-plus-noise ratio for fair target detection, subject to the constraints of the communication quality of service, different BD-RIS architectures and power budget. To solve the non-convex and non-smooth max-min problem, a general solution based on the alternating direction method of multipliers is provided for all considered BD-RIS architectures. Numerical simulations validate the efficacy of the proposed algorithm and show the superiority of the BD-RIS aided DFRC system in terms of both communication and sensing compared to conventional RIS aided DFRC.