Beam alignment is a critical bottleneck in millimeter wave (mmWave) communication. An ideal beam alignment technique should achieve high beamforming (BF) gain with low latency, scale well to systems with higher carrier frequencies, larger antenna arrays and multiple user equipments (UEs), and not require hard-to-obtain context information (CI). These qualities are collectively lacking in existing methods. We depart from the conventional codebook-based (CB) approach where the optimal beam is chosen from quantized codebooks and instead propose a grid-free (GF) beam alignment method that directly synthesizes the transmit (Tx) and receive (Rx) beams from the continuous search space using measurements from a few site-specific probing beams that are found via a deep learning (DL) pipeline. In realistic settings, the proposed method achieves a far superior signal-to-noise ratio (SNR)-latency trade-off compared to the CB baselines: it aligns near-optimal beams 100x faster or equivalently finds beams with 10-15 dB better average SNR in the same number of searches, relative to an exhaustive search over a conventional codebook.