Text-free diffusion inpainting using reference images for enhanced visual fidelity - Kim, Beomjo; Sohn, Kyung Ah (Pattern Recognition Letters, 2024)
This paper presents a novel approach to subject-driven image generation that addresses the limitations of traditional text-to-image diffusion models. Our method generates images using reference images without relying on language-based prompts. We introduce a visual detail preserving module that captures intricate details and textures, addressing overfitting issues associated with limited training samples. The model's performance is further enhanced through a modified classifier-free guidance technique and feature concatenation, enabling the natural positioning and harmonization of subjects within diverse scenes. Quantitative assessments using CLIP, DINO and Quality scores (QS), along with a user study, demonstrate the superior quality of our generated images. Our work highlights the potential of pre-trained models and visual patch embeddings in subject-driven editing, balancing diversity and fidelity in image generation tasks. Our implementation is available at https://github.com/8eomio/Subject-Inpainting.
Keywords: Diffusion model; Free diffusion; Image diffusion; Image generations; Image Inpainting; Image manipulation; Inpainting; Reference image; Subject-driven generation; Visual fidelity; Diffusion models; Image generation; Image inpainting; Image manipulation; Subject-driven generation