Proteins structure prediction has long been a grand challenge over the past 50 years, owing to its board scientific and application interests. There are two major types of modelling algorithm, template-free modelling and template-based modelling, which is suitable for easy prediction tasks, and is widely adopted in computer aided drug discoveries for drug design and screening. Although it has been several decades since its first edition, the current template-based modeling approach suffers from two important problems: 1) there are many missing regions in the template-query sequence alignment, and 2) the accuracy of the distance pairs from different regions of the template varies, and this information is not well introduced into the modeling. To solve the two problems, we propose a structural optimization process based on template modelling, introducing two neural network models predict the distance information of the missing regions and the accuracy of the distance pairs of different regions in the template modeling structure. The predicted distances and residue pairwise specific accuracy information are incorporated into the potential energy function for structural optimization, which significantly improves the qualities of the original template modelling decoys.