Assignment #5 - Cat Photo Editing

Motivation

Edit the style of input image using pretrained model.

Part 1

In the styleGAN paper, the style is sampled from the latent space and combined to the synthesis network through AdaIN, where each feature map $x_i$$x_i$ is normalized separately, then scaled and biased using the corresponding $y_{s,i}$$y_{s,i}$ and $y_{b,i}$$y_{b,i}$.

There are three choices of latent space:

• z -- Gaussian Noise
• w -- $\mathit{M}\mathit{a}\mathit{p}\mathit{p}\mathit{i}\mathit{n}\mathit{g}\mathit{N}\mathit{e}\mathit{t}\mathit{w}\mathit{o}\mathit{r}\mathit{k}\mathit{(}\mathit{z}\mathit{)}$$\it{MappingNetwork(z)}$
• w+ -- use different w for different layers

We should initialize the latent first, then pass the latent through a pretrained Generator, use the optimization-based method to draw the image near the target while keeping the image style faithful to the pretrained model.

The choice of latent space, loss weights, optimizer and data sample leads to results in different quality.

LBFGS seems faster than the Adam optimizer. Since the sample only changes fast in the first several steps but remains roughly the same later in some cases.

The results from z/w/w+latent space can all be faithful to the target(content image) in some range, and realistic. In practice, it is harder to find a good set of loss weight in w+. Here shows the results using optimal parameters among all trys.

 

Part 2

In part2, we are trying to add content and style to sketch image, while preserving the contour information indicated by the user given sketch.

The sketch image(RGBA) serves as the input, and a corresponding mask(0,1) is generated according to its alpha channel.

We still use two terms of loss here:

• the perceptual loss $z^{\ast }=argmi{n}_z\sum _i||f_i(G(z))-v_i|{|}_1$$z^* = argmin_z \sum_i ||f_i(G(z))-v_i||_1$
• the Lp loss $z^{\ast }=argmi{n}_z||M\ast G(z)-M\ast S|{|}_1$$z^* = argmin_z ||M * G(z) - M * S||_1$

The results using latents from different spaces not variate so much under coarse sketch condition, but obvious different in dense sketch.

In the code implementation, these two losses all support mask. But using mask in feature space(percetual loss) seems weired, because after the conv extraction, the feature belongs to sketch might not be staying at the same place. For denser stroke, this effect might be alleviated.

Part 3

In this part, we need to add noise to the encoded input sketch $x_{t-1}\to x_t$$x_{t-1} \rightarrow x_{t}$, then denoise from the noisy latent using parameters stored in the pretrained model, which can calculate the mean and variance of each step's distribution $x_t\to x_{t-1}$$x_{t} \rightarrow x_{t-1}$, plus the cond and uncond guidance. I use the cfg rate=7.

The sample step $N$$N$ is recommended to be 500-700, because the DDPM pretraining uses 1000 steps between the encoded real image and the noise. Thus a $N$$N$ less than 1000 will reserve the low level features of the sketch, which enables the content of sketch to be kept.

(My drawing of Grumpy Cat doesn't look grumpy emm)