GAN Photo Editing

Introduction

In this assignment, we will try to create an image based on a sketch. First, we will invert a pre-trained generator to find a latent variable that closely reconstructs the given real image. In the second part, we will take a hand-drawn sketch and generate an image that fits the sketch accordingly.

Generator inversion

Here is the target image.

Here are the results of using different weights of perceptual loss using vanilla gan (250 optimization step).

0 perceptual loss weight	0.5 perceptual loss weight	1 perceptual loss weight

Here are the results of using different weights of perceptual loss using vanilla gan (1000 optimization step).

0 perceptual loss weight	0.5 perceptual loss weight	1 perceptual loss weight

Due to the limitation of vanilla gan, the output is not close to the target and clear regardless of perceptual weight and optimization steps. Here are the results of using different weights of perceptual loss using stylegan and w latent space (250 optimization step).

0 perceptual loss weight	0.5 perceptual loss weight	1 perceptual loss weight

Here are the results of using different weights of perceptual loss using stylegan and w latent space (1000 optimization step).

0 perceptual loss weight	0.5 perceptual loss weight	1 perceptual loss weight

The quality of image is greatly improved compared to vanilla gan. However, there are only slight changes with respect to optimization steps and perceptual loss weight. Personally I would use the perceptual weight of 1. It is also worth noting that the speed of optimization is slower due to more complex network structure. Here are the results of using different weights of perceptual loss using stylegan and w+ latent space (250 optimization step).

0 perceptual loss weight	0.5 perceptual loss weight	1 perceptual loss weight

Here are the results of using different weights of perceptual loss using stylegan and w+ latent space (1000 optimization step).

0 perceptual loss weight	0.5 perceptual loss weight	1 perceptual loss weight

Once again, the differences are subtle. It appears that using w+ space helps reconstruct certain aspects of the original image, like the color hue of the eyes. It is also worth noting that the optimization of w+ space is not as stable, as there is a slight chance that the output image does not resemble the target image.

Interpolation

Here are some interpolation results.

source image	target image	interpolation
source image	target image	interpolation
source image	target image	interpolation

Most interpolation results are very smooth. There are certain exceptions like the 3rd interpolation result. Because the color and eye color of the cats are too different, the interpolation result is not convincing.

Sketch 2 Image

The sketch 2 image process is almost identical to the reconstruction process, except it is trying to reconstruct the sketch under the filter of mask. Note that I did not use any regularization techniques.
Here are some of the sketch2image attempts.

</tr>

</tr>

</tr>

</tr>

</tr>

sketch	output image
sketch	output image
sketch	output image
sketch	output image
sketch	output image

The method appears to turn the sketch to image quite successfully, preserving many features like shape of the cats face and some color features.