Automatic defect recognition is one of the research hotspots in steel production, but most of current mehthods mainly extract features manually and use machine learning classifiers to recognize, which cannot tackle the situation,where there are few data available to train, and confine to certain scene. Therefore, in this paper, a new approach is proposed which consists of part of pretrained VGG16 as feature extractor and a new CNN neural network as classifier to recognize the defect of steel strip surface based on the feature maps created by the feature extractor. We obtain 96% accuracy with train set which only contain 10 images in each class, which is much better than previous method.