One of the critical biotic stress factors paddy farmers face is diseases caused by bacteria, fungi, and other organisms. These diseases affect plants' health severely and lead to significant crop loss. Most of these diseases can be identified by regularly observing the leaves and stems under expert supervision. In a country with vast agricultural regions and limited crop protection experts, manual identification of paddy diseases is challenging. Thus, to add a solution to this problem, it is necessary to automate the disease identification process and provide easily accessible decision support tools to enable effective crop protection measures. However, the lack of availability of public datasets with detailed disease information limits the practical implementation of accurate disease detection systems. This paper presents Paddy Doctor, a visual image dataset for identifying paddy diseases. Our dataset contains 13,876 annotated paddy leaf images across ten classes (nine diseases and normal leaf). We benchmarked the Paddy Doctor using a Convolutional Neural Network (CNN) and two transfer learning approaches, VGG16 and MobileNet. The experimental results show that MobileNet achieves the highest classification accuracy of 93.83\%. We release our dataset and reproducible code in the open source for community use.