Saliency Constrained Arbitrary Image Style Transfer using SIFT and DCNN

This paper develops a new image synthesis approach to transfer an example image (style image) to other images (content images) by using Deep Convolutional Neural Networks (DCNN) model. When common neural style transfer methods are used, the textures and colors in the style image are usually transferred imperfectly to the content image, or some visible errors are generated. This paper proposes a novel saliency constrained method to reduce or avoid such effects. It first evaluates some existing saliency detection methods to select the most suitable one for use in our method. The selected saliency detection method is used to detect the object in the style image, corresponding to the object of the content image with the same saliency. In addition, aim to solve the problem that the size or resolution is different in the style image and content, the scale-invariant feature transform is used to generate a series of style images and content images which can be used to generate more feature maps for patches matching. It then proposes a new loss function combining the saliency loss, style loss and content loss, adding gradient of saliency constraint into style transfer in iterations. Finally the source images and saliency detection results are utilized as multichannel input to an improved deep CNN framework for style transfer. The experiments show that the saliency maps of source images can help find the correct matching and avoid artifacts. Experimental results on different kind of images demonstrate that our method outperforms nine representative methods from recent publications and has good robustness.