Establishing and maintaining millimeter-wave (mmWave) links is challenging due to the changing environment and the high sensibility of mmWave signal to user mobility and channel conditions. MmWave link configuration problems often involve a search for optimal system parameter under environmental uncertainties, from a finite set of alternatives that are supported by the system hardware and protocol. For example, beam sweeping aims at identifying the optimal beam(s) for data transmission from a discrete codebook. Selecting parameters such as the beam sweeping period and the beamwidth are crucial to achieving high overall system throughput. In this article, we motivate the use of the multi-armed bandit (MAB) framework to intelligently search out the optimal configuration when establishing the mmWave links. MAB is a reinforcement learning framework that guides a decision-maker to sequentially select one action from a set of actions. As an example, we show that within the MAB framework, the optimal beam sweeping period, beamwidth, and beam directions could be dynamically learned with sample-computational-efficient bandit algorithms. We conclude by highlighting some future research directions on enhancing mmWave link configuration design with MAB.