As a cost-effective alternative, hybrid analog and digital beamforming architecture is a promising scheme for millimeter wave (mmWave) system. This paper considers two hybrid beamforming architectures, i.e. the partially-connected and fully-connected structures, for mmWave dual-function radar communication (DFRC) system, where the transmitter communicates with the downlink users and detects radar targets simultaneously. The optimization problems are formulated by minimizing a weighted summation of radar and communication performance, subject to constant modulus and power constraints. To tackle the non-convexities caused by the two resultant problems, effective Riemannian optimization algorithms are proposed. Specifically, for the fully-connected structure, a manifold algorithm based on the alternating direction method of multipliers (ADMM) is developed. While for the partially-connected structure, a low-complexity Riemannian product manifold trust region (RPM-TR) algorithm is proposed to approach the near-optional solution. Numerical simulations are provided to demonstrate the effectiveness of the proposed methods.