This paper proposes a method for estimating the head pose from a single image. This estimation uses a neural network (NN) obtained in two stages. In the first stage, we trained the base NN, which has one regression head and four regression via classification (RvC) heads. We build the ensemble of offsets using small offsets of face bounding boxes. In the second stage, we perform knowledge distillation (KD) from the ensemble of offsets of the base NN into the final NN with one RvC head. On the main test protocol, the use of the offset ensemble improves the results of the base NN, and the KD improves the results from the offset ensemble. The KD improves the results by an average of 7.7\% compared to the non-ensemble version. The proposed NN on the main test protocol improves the state-of-the-art result on AFLW2000 and approaches, with only a minimal gap, the state-of-the-art result on BIWI. Our NN uses only head pose data, but the previous state-of-the-art model also uses facial landmarks during training. We have made publicly available trained NNs and face bounding boxes for the 300W-LP, AFLW, AFLW2000, and BIWI datasets. KD-ResNet152 has the best results, and KD-ResNet18 has a better result on the AFLW2000 dataset than any previous method.