Because of the advantages of easy deployment, low cost and non-contact, computer vision-based structural displacement acquisition technique has received wide attention and research in recent years. However, the displacement field acquisition of large-scale structures is a challenging topic due to the contradiction of camera field of view and resolution. This paper presents a large-scale structural displacement field calculation framework with integrated computer vision and physical constraints using only one camera. Firstly, the full-field image of the large-scale structure is obtained by processing the multi-view image using image stitching technique; secondly, the full-field image is meshed and the node displacements are calculated using an improved template matching method; and finally, the non-node displacements are described using shape functions considering physical constraints. The developed framework was validated using a scaled bridge model and evaluated by the proposed evaluation index for displacement field calculation accuracy. This paper can provide an effective way to obtain displacement fields of large-scale structures efficiently and cost-effectively.