Non-blind deblurring methods achieve decent performance under the accurate blur kernel assumption. Since the kernel error is inevitable in practice, ringing artifacts are often introduced by non-blind deblurring. Recently, semi-blind deblurring methods can handle kernel uncertainty by introducing the prior of the kernel (or induced) error. However, how to design a suitable prior for the kernel (or induced) error remains challenging. Hand-crafted prior, incorporating domain knowledge, generally performs well but may lead to poor performance when kernel (or induced) error is complex. Data-driven prior, which excessively depends on the diversity and abundance of training data, is vulnerable to out-of-distribution blurs and images. To address this challenge, we suggest a data-free deep prior for the kernel induced error (termed as residual) expressed by a customized untrained deep neural network, which allows us to flexibly adapt to different blurs and images in real scenarios. By organically integrating the respective strengths of deep priors and hand-crafted priors, we propose an unsupervised semi-blind deblurring model which recovers the latent image from the blurry image and inaccurate blur kernel. To tackle the formulated model, an efficient alternating minimization algorithm is developed. Extensive experiments demonstrate the superiority of the proposed method to both data-driven prior and hand-crafted prior based methods in terms of the image quality and the robustness to the kernel error.