Alert button

Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion

Aug 02, 2023
Zixuan Ni, Longhui Wei, Jiachen Li, Siliang Tang, Yueting Zhuang, Qi Tian

Figure 1 for Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion
Figure 2 for Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion
Figure 3 for Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion
Figure 4 for Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion

Share this with someone who'll enjoy it:

Owing to the unrestricted nature of the content in the training data, large text-to-image diffusion models, such as Stable Diffusion (SD), are capable of generating images with potentially copyrighted or dangerous content based on corresponding textual concepts information. This includes specific intellectual property (IP), human faces, and various artistic styles. However, Negative Prompt, a widely used method for content removal, frequently fails to conceal this content due to inherent limitations in its inference logic. In this work, we propose a novel strategy named \textbf{Degeneration-Tuning (DT)} to shield contents of unwanted concepts from SD weights. By utilizing Scrambled Grid to reconstruct the correlation between undesired concepts and their corresponding image domain, we guide SD to generate meaningless content when such textual concepts are provided as input. As this adaptation occurs at the level of the model's weights, the SD, after DT, can be grafted onto other conditional diffusion frameworks like ControlNet to shield unwanted concepts. In addition to qualitatively showcasing the effectiveness of our DT method in protecting various types of concepts, a quantitative comparison of the SD before and after DT indicates that the DT method does not significantly impact the generative quality of other contents. The FID and IS scores of the model on COCO-30K exhibit only minor changes after DT, shifting from 12.61 and 39.20 to 13.04 and 38.25, respectively, which clearly outperforms the previous methods.

* ACM MM 2023  
View paper onarxiv icon

Share this with someone who'll enjoy it: