Alert button
Picture for Chengming Feng

Chengming Feng

Alert button

Controlling Neural Style Transfer with Deep Reinforcement Learning

Sep 30, 2023
Chengming Feng, Jing Hu, Xin Wang, Shu Hu, Bin Zhu, Xi Wu, Hongtu Zhu, Siwei Lyu

Figure 1 for Controlling Neural Style Transfer with Deep Reinforcement Learning
Figure 2 for Controlling Neural Style Transfer with Deep Reinforcement Learning
Figure 3 for Controlling Neural Style Transfer with Deep Reinforcement Learning
Figure 4 for Controlling Neural Style Transfer with Deep Reinforcement Learning

Controlling the degree of stylization in the Neural Style Transfer (NST) is a little tricky since it usually needs hand-engineering on hyper-parameters. In this paper, we propose the first deep Reinforcement Learning (RL) based architecture that splits one-step style transfer into a step-wise process for the NST task. Our RL-based method tends to preserve more details and structures of the content image in early steps, and synthesize more style patterns in later steps. It is a user-easily-controlled style-transfer method. Additionally, as our RL-based model performs the stylization progressively, it is lightweight and has lower computational complexity than existing one-step Deep Learning (DL) based models. Experimental results demonstrate the effectiveness and robustness of our method.

* Accepted by IJCAI 2023. The contributions of Chengming Feng and Jing Hu to this paper were equal. arXiv admin note: text overlap with arXiv:2309.13672 
Viaarxiv icon

Deep Reinforcement Learning for Image-to-Image Translation

Sep 24, 2023
Xin Wang, Ziwei Luo, Jing Hu, Chengming Feng, Shu Hu, Bin Zhu, Xi Wu, Siwei Lyu

Figure 1 for Deep Reinforcement Learning for Image-to-Image Translation
Figure 2 for Deep Reinforcement Learning for Image-to-Image Translation
Figure 3 for Deep Reinforcement Learning for Image-to-Image Translation
Figure 4 for Deep Reinforcement Learning for Image-to-Image Translation

Most existing Image-to-Image Translation (I2IT) methods generate images in a single run of a deep learning (DL) model. However, designing such a single-step model is always challenging, requiring a huge number of parameters and easily falling into bad global minimums and overfitting. In this work, we reformulate I2IT as a step-wise decision-making problem via deep reinforcement learning (DRL) and propose a novel framework that performs RL-based I2IT (RL-I2IT). The key feature in the RL-I2IT framework is to decompose a monolithic learning process into small steps with a lightweight model to progressively transform a source image successively to a target image. Considering that it is challenging to handle high dimensional continuous state and action spaces in the conventional RL framework, we introduce meta policy with a new concept Plan to the standard Actor-Critic model, which is of a lower dimension than the original image and can facilitate the actor to generate a tractable high dimensional action. In the RL-I2IT framework, we also employ a task-specific auxiliary learning strategy to stabilize the training process and improve the performance of the corresponding task. Experiments on several I2IT tasks demonstrate the effectiveness and robustness of the proposed method when facing high-dimensional continuous action space problems.

Viaarxiv icon