Caption: A research poster titled "Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?" is prominently displayed at a conference. The poster, associated with several academic and research institutions, explores the effectiveness of perturbation-based protection techniques to prevent the exploitation of personal images by Stable Diffusion models. It systematically evaluates various protection methods, discusses the limitations of current approaches, and introduces a new method called GrID*Pure for removing protective perturbations while preserving image details. The poster includes several diagrams, charts, and image comparisons to support the findings and is presented at a booth numbered 17, adjacent to booth 16. Participants and attendees can be seen examining the detailed content of the poster against the backdrop of a well-lit conference hall. (Note: Individual names and institutions have been deliberately included as they appear in the image and are relevant to the content described.) Text transcribed from the image: 16 ICT Drexel UNIVERSITY protected data person TV style DreamBooth Textual Inversion LORA Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion? Zhengyue Zhao12, Jinhao Duan³, Kaidi Xu, Chenan Wang, Rui Zhang, Zidong Du, Qi Guo, Xing Hu¹ Institute of Computing Technology, Chinese Academy of Sciences 3 Drexel University SUCCESSFUL PROTECTION protected generation 2 University of Chinese Academy of Sciences Effectiveness of Perturbation-based Protection The effectiveness of perturbation-based protection varies significantly depending on different fine-tuning methods. These perturbations rely on attacks against the text encoder and yield smaller benefits for methods that don't require fine-tuning text encoder. The protection method is sensitive to the proportion of images being protected. Perturbing images with a small ratio is insufficient to provide effective protection. ➤Natural transformations such as Gaussian blur and JPEG can significantly reduce the protection effectiveness. 4 stride-128 purify transform different settings Text Image FAILED PROTECTION Stable Diffusion protective perturbation unprotected data fine-tuning unprotected generation 5 GOP Unconditional Diffusion Model (256x256) CVPR SEATTLE, WA JUNE 17-21, 2024 " Average Merge diffusion denoising mall Introduction clean data ➤ Stable Diffusion has established itself as a foundation model in generative Al artistic applications while also giving rise to issues like facial privacy forgery and artistic copyright infringement. Recent studies have explored the addition of imperceptible adversarial perturbations to personal images to prevent potential unauthorized exploitation and infringements when these data are used for fine-tuning Stable Diffusion models. This paper systematically evaluates the effectiveness of recent protective perturbations within a practical threat model. Results show that these approaches may not be sufficient to safeguard image privacy and copyright. ➤ A simple yet effective purification, GrIDPure, is introduced to remove all these protective perturbations while preserving the structure of images. Text2img DreamBooth Text. Inversion Text2img DreamBooth Text. Inversion 150 04 0.6 protection ratio FD-4263 HD AdvOM 0.6 FID AntDB 0.50 CUP AdvOM CLIP AntiDB 040 FD-125 no-112 no-2753 +1280 0.3 Bypass Protective Perturbation via Purification Adversarial purification methods such as DiffPure have shown their ability to remove adversarial perturbations in image classification tasks. This paper demonstrates that this Sota purification method also does well in purifying protected images into learnable images even under carefully designed adaptive attacks. Compared with the purification of adversarial examples for image classification, removing protective perturbation should not reduce the image's quality. DiffPure may disrupt the details of complex paintings, which affects its practical utility. 512x512 Nx 256x256 256x256 High resolution 256x256 Low resolution 256x256 512x512 High resolution GrIDPure: Removing Protection and Preserving Details ➤GrIDPure includes three key steps: (1) The protective high-resolution image is first divided into multiple grids, ensuring that each part of the image overlaps with at least two grids. (2) Each grid is then purified with SDEdit, employing an unconditional diffusion model with small steps. (3) The purified grids are merged back into a high-resolution image, with any overlapping parts being averaged during the merging process. Compared with Vallina DiffPure, GrIDPure can better preserve the details of artworks while successfully removing all kinds of protective perturbations, which means protective perturbations can be easily bypassed. The applicability of GrIDPure reveals the fragility of protecting personal images from being learned by Stable Diffusion via adversarial perturbations. Don't Train TE Text2img LORA Custom Diff Text2imp LORA DreamBooth Ground truth 17