Assignment #5 – Image Synthesis Project 2

Table of Contents

EXPLANATION

In this assignment, we will implement a few different techniques that require you to manipulate images on the manifold of natural images. First, we will invert a pre-trained generator to find a latent variable that closely reconstructs the given real image. In the second part of the assignment, we will take a hand-drawn sketch and generate an image that fits the sketch accordingly.

We will be using a pre-trained network for doing so, we will be working with vanilla GAN and style gan with latent space of z, w, and w+ as mentioned in the Image2Style++ paper.

Figure 0: Results after interpolation of cats

Part 1: Inverting the Generator

We implemented content-space loss and optimize a random noise with respect to the content loss only. The content loss between two images can be provided as below:

Figure 1: Content Image Loss

We choose to consider the output manifold of a trained generator as close to the natural image manifold. So, we can set up the following nonconvex optimization problem as below:

Figure 2: Non-convex optimization problem

As this is a nonconvex optimization problem where we can access gradients, we can attempt to solve it with any first-order or quasi-Newton optimization method (e.g., LBFGS). One issue here is that these optimizations can be both unstable and slow. We try running the optimization from many random seeds and taking a stable solution with the lowest loss as your final output.

Results

Round: 250

Round: 500

Round: 750

Round: 1000

Figure 3: Vanilla GAN Inversion

Round: 250

Round: 500

Round 750

Round 1000

Figure 4: Style GAN Loss with Latent space as w

Round 250

Round 500

Round: 750

Round: 1000

Figure 5: Style GAN with latent space w+

Round: 250

Round: 500

Round: 750

Round: 1000

Figure 6: Vanilla GAN with latent space z using perceptual loss + L1 loss

Round: 250

Round: 500

Round: 1000

Figure 7: Style gan with latent space w+ using style loss along with perceptual & L1 loss

Part 2: Interpolate your Cats

Here we use a convex combination of inverses to interpolate between images. We experiment with different generative models and different latent spaces (latent code z, w space, and w+ space).

Results

Style GAN in Z space

Source

Destination

Interpolation

Source

Destination

Interpolation

Source

Destination

Interpolation

Style GAN in W space

Source

Destination

Interpolation

Source

Destination

Interpolation

Source

Destination

Interpolation

Style GAN in W+ space

Source

Destination

Interpolation

Source

Destination

Interpolation

Part 3: Scribble to Image

Next, we would like to constrain our image in some way while having it look realistic. This constraint could be color scribble constraints as we initially tackle this problem, but could be many other things as well. We will initially develop this method in general and then talk about color scribble constraints in particular. To generate an image subject to constraints, we solve a penalized nonconvex optimization problem. We’ll assume the constraints are of the form: