In my project, I implemented techniques to manipulate images on the manifold of natural images.
First, I inverted a pre-trained generator to find a latent variable that closely reconstructs a given real image.
This involved solving a nonconvex optimization problem using a loss function and a trained generator,
where I tried out different losses and optimization methods to find the best solution.In the second part of the project,
I interpolated through images by combining latent images for two images and generating a new image.
Finally, I generated an image subject to constraints by solving a penalized nonconvex optimization problem,
where I used color scribble constraints to fill in details of a hand-drawn image. I reduced the equation under
the constraints to find the optimal solution using the Hadamard product, mask, and sketch.
I inverted a pre-trained generator to find a latent variable that closely reconstructs a given real image.
This was done by solving a nonconvex optimization problem using a loss function and a trained generator, and I tried out
different losses and optimization methods to find the best solution. My results showed that using a combination of perceptual
and L1 losses with the LBFGS optimization method gave me the most stable and efficient solution,
with a low reconstruction loss for the generated image.
| Original Image | Vanilla GAN z | StyleGAN z | StyleGAN w | StyleGAN w+ |
|
|
|
|
|
I tried different combinations of VGG-19 layers for perceptual loss and the best result were generated
while using 'conv_1','conv_2','conv_5','conv_9' and 'conv_13' layers. After much experimenting I used
the default 0.01 weight for the perceptual loss and 10 for pixel level L1 loss. I compared the outputs of
vnailla GAN and styleGAN with different latent spaces (z, w and w+). The optimized result was reached at
the 6000th iteration and more optimization did not result in better image quality.
I interpolated through images by combining latent images for two images and generating a new image.
I experimented with different generative models and different latent spaces (latent code z, w space, and w+ space)
to see which worked best. My results showed that using the StyleGAN2 model and the w+ space gave me
the most visually pleasing results, with smooth transitions between images and realistic features.
| Image A | Interpolations | Image B |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Above gif represents the interpolation between two latent spaces using StyleGAN and w+ space to embedded two images into the latent space.
The interpolated result smoothly translates the semantic details between Image A and Image B.
Used color scribble constraints to fill in details of a cat image, and reduced the equation under the constraints
to find the optimal solution using the Hadamard product, mask, and sketch. My results showed that the model was able to accurately
fill in the details of the image while maintaining it appearance, and the color scribble constraints helped to guide the generation process.
| Sketchs | Reneration Images |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |