Final Project - Text-to-Style Reconstruction for Diffusion Style Transfer

Image embedding generated by CLIP image encoder can have style parts and content parts in the CLIP embedding space. Similarly, text embedding obtained from CLIP text encoder should have style parts. Assuming it, the method proposed in ITstyler [Bai et al., 2023] used MLP networks to map from CLIP embedding space to style space encoded by pre-trained VGG network [Simonyan and Zisserman, 2014]. While the mean and variance of the output of intermediate layer in VGG are considered as the style representation of images and used in AdaIN-like architecture in the model in ITstyler, in this project I used Gram matrix representation of styles as proposed in [Gatys et al., 2015] so that the style representation can be used in guidance in diffusion model. Therefore, I trained Mapping Network φ that maps CLIP embedding to the Gram matrix representation of style using following loss function:

where Gram is the Gram matrix representation of images with VGG and the mapping network learned how to convert CLIP encoding to style encoding. At inference time, text inputs instead of image inputs are used to get Gram matrix representation. A figure below describes the architecture of style encoding networks and training of them.

I used pre-trained diffusion models since [Yang et al., 2023] showed that the pre-trained model with guidance of both contents and style can achieve style transfer tasks. DDPM [Ho et al., 2020] directly samples xt from original image x0 by adding Gaussian noise with βt ∈ (0, 1) at time t ∈ [1, ..., T ] using following formula

where ε ∼ N (0, I), αt = 1 − βt, and ̄αt is products of αi. The reverse sampling process to generate a clean image is obtained by applying following backward diffusion step repeatedly:

where the network εθ (xt, t) returns the predicted noise. Then we used loss function defined as follows:

L_content and L_style represent content loss and style loss respectively. This loss function is used as guidance in the sampling process to get the denoised image as follows:

Style loss is basically L2 loss of the Gram matrix of text-style embedding and VGG style embedding of generated images:

where p_target is a target prompt.

The content loss consists of Patch-wise Zero-shot Contrastive loss (ZeCon) [Yang et al., 2023], perceptual loss using VGG, and pixel-lebel MSE loss. Patch-wise ZeCon loss [Yang et al., 2023] helps the model keep semantic content in the denoising process. Intuitively, ZeCon loss will make patches in the same location of ˆx0,t and x0 similar and patches in different locations of ˆx0,t and x0 different. It is defined mathematically as follows:

All losses are weighted when it is used and weights are tuned as hyperparameters.

To show how much the mapping network can extract style representation from texts, first I performed style reconstruction of texts. Specifically, I tried to reconstruct images that minimize L_style with text inputs. The results of the reconstruction and the prompts used are shown below. As we can see, colors used in the reconstructed images match the input prompts but the images are blur and noisy. Probably it is because of the error of the mapping network. Also, the model failed to learn texture since all reconstructed images do not have differences in texture.

Also, the results of neural style transfer with content images and style texts (similar to what we did in HW4) are shown below. It successfully transferred the style of the text inputs but the images generated are still noisy because the Gram matrix from the mapping network is noisy.

As we saw, while the direct usage of the mapping network in the style reconstruction returns successful style transfer from text to image, the images generated are blur and noisy. The next idea to utilize this style representation of texts is diffusion style transfer. The logic behind this idea is that the knowledges of the pre-trained diffusion models can be leveraged to synthesize less blur or noisy images and transfer styles using the guidance at the same time.

The results of the diffusion style transfer are shown below. Compared to the neural style transfer, it failed to transfer styles from text inputs. The reason might be the difference of the image generation and optimization process. While the pixel values are directly optimized to minimize the style loss in the neural style transfer, the predicted noises are guided to reduce the style loss. The guidance might not be enough to transfer styles especially when the style is defined as the Gram matrix.

Also, the following images show the effects of the weight on the style loss. When the weights on the style loss increase, the content of the original images tends to get lost and some random patterns show up. Also, if the weights of the loss are too high it caused mathematical unstability and the model failed to compute the gradient of the loss function and ended up with returning all black images.

The comparisons with the model using CLIP loss [Yang et al., 2023] are shown below. The possible reason is that our model failed to learn texture of images and to replicate textures from text inputs.