Millimeter wave (mmWave) communication is regarded as a key enabled technology for the future wireless communication to satisfy the requirement of Gbps transmission rate and address the problem of spectrum shortage. Directional transmission used to combat the large pathloss of mmWave communications helps to realize the device-to-device (D2D) communication in ultra-dense networks. In this paper, we consider the problem of joint beam selection and link activation across a set of communication pairs in ultra-dense D2D mmWave networks. The resulting optimization problem is formulated as an integer programming problem that is nonconvex and NP-hard problem. Consequently, the global optimal solution, even the local optimal solution, cannot be generally obtained. To overcome this challenge, we resort to design a deep learning architecture based on graphic neural network to finish the joint beam selection and link activation, called as GBLinks model, with taking into account the network topology information. We further present an unsupervised Lagrangian dual learning framework to train the parameters of GBLinks. Numerical results show that the proposed GBLinks model can converges to a stable point with the number of iterations increases, in terms of the average sum rate. It also shows that GBLinks can reach near-optimal solution through comparing with the exhaustively search in small-scale D2D mmWave networks and outperforms selfish beam selection strategy with activating all links.