Assignment #4 - Neural Style Transfer

Sanil Pande (spande)

PART 1: Content Reconstruction

The following content and style images have been used for experiments:

The content reconstruction using loss from different layers is as follows:

Conv1                         Conv2                         Conv3                         Conv4

Effect of different layers - while the difference is not significant in the above images, it could be because the layers are too close together. On optimizing for later layers,  (conv14 for example) the reconstruction is extremely noisy and does not resemble the content image.

Initializing for only content with the following image gives the following images:

Original:

Reconstruction 1:

Reconstruction 2:

The reconstructed images look very similar to the original, except for some noise that can be noticed.

PART 2: Style Reconstruction

Initializing for only style [conv1, conv2, conv3, conv4, conv5] gives the following images:

Reconstruction 1:                                        Reconstruction 2:

Effect of different layers - while the difference is not significant in the above images, it could be because the layers are too close together. On optimizing for later layers,  (conv11-15 for example) the reconstruction is less blocky (much finer style) but the content is not reconstructed at all.

PART 3: Style Transfer

Implementation - the content loss is taken as the euclidean distance between filter responses at intermediate layer (conv4 specifically). The style loss is the euclidean distance between the normalized Gram matrices of the activations at intermediate layers. The weight for the style loss has to be much higher than the weight for the content loss, and this works quite well for when the image to be optimized is initialized as the content image.

                                                Style Images

Content images                

Reconstruction from the random image vs content image:

We can see that the image initialized from random noise very imperceptibly resembles some of the structure of the input content image, but that the one initialized from the content image itself is

Style Transfer on my own images - I have implemented style transfer on the following 2 images;

Style Transfer with the Frida Kahlo portrait -

Content  #2

Style  #2

Result #2