Bengali handwritten digit recognition can be done using different image classification techniques. But the images of handwritten digits are different from natural images as the orientation of a digit as well as similarity of features of different digits are important. On the other hand, deep convolutional neural networks are achieving huge success in computer vision problems, especially in image classification. This BDNet is a densely connected deep convolutional neural network model based on state-of-the-art algorithm DenseNet to classify Bengali handwritten numeral digits. The BDNet has end-to-end trained using ISI Bengali handwritten numeral dataset with 5-fold cross-validation. The BDNet has achieved a test accuracy of 99.65% (baseline was 99.40%) on test data of ISI Bengali handwritten numerals. The trained model also gives 97.50% on own created dataset (which are not used during training). That is, this model gives a 41.66% error reduction compared to the previous state-of-the-art model. Codes, trained model and own dataset available at: https://github.com/Sufianlab/BDNet.