"Exploiting the Signal-Leak Bias in Diffusion Models" at WACV 2024

Our paper "Exploiting the Signal-Leak Bias in Diffusion Models" was accepted and presented at the Winter Conference on Applications of Computer Vision (WACV 2024). Diffusion models generate images by gradually denoising a random noise sample. Our work provides control over the generated images by modifying this initial noise sample, leading for instance to better style adaptation results and more colorful images.

There is a bias in the inference pipeline of most diffusion models. This bias arises from a signal leak whose distribution deviates from the noise distribution, creating a discrepancy between training and inference processes. We demonstrate that this signal-leak bias is particularly significant when models are tuned to a specific style, causing sub-optimal style matching. Recent research tries to avoid signal leakage during training. We instead show how we can exploit this signal-leak bias in existing diffusion models to allow more control over the generated images. This enables us to generate images with more varied brightness, and images that better match a desired style or color. By modeling the distribution of the signal leak in the spatial frequency and pixel domains, and including a signal leak in the initial latent, we generate images that better match expected results without any additional training.

Project website: https://ivrl.github.io/signal-leak-bias/

WACV proceedings: https://openaccess.thecvf.com/content/WACV2024/html/Everaert_Exploiting_the_Signal-Leak_Bias_in_Diffusion_Models_WACV_2024_paper.html