16-726 Project 3, Jason Xu

Overview

This project aims to implement the layers and train two machine learning networks, DCGAN and CycleGAN.  

Padding for Discrimintor Layers

• We denote n[l] to be the dimensions of the current layer, p to be padding amount, f to be convolution kernel size, and s to be stride. We have:
• n[l] = (n[l-1]+2p-f[l])/s+1
• We rearrange this equation to get p, which is:
• p = (s(n[l]-1)-n[l-1]+f[l])/2
• We then let s=2, f=4 and n[l-1] = n, which means n[l] = n/2, which is the case for the DCDiscriminator. We get:
• p = (2(n/2-1)-n+4)/2
• = (n-2-n+4)/2
• = 1
• Since the relationship between n[l-1] and n[l] applies to all layers, all layers have padding of 1.

Differentiable Augumentation

Differentiable augumentation is implemented using random edits to brightness, contrast, saturation and random translation and cutout for the real and fake images. This likely prevents overfitting on training the discriminator, enabling better results.

When not using differentiable augumentation, although discriminator loss drops quicker, generator loss does not drop at all. The reuslts returned (below) are also noticeably worse.

Without differentiable augumentation

With differentiable augumentation

DCGAN Results

Discriminator and generator training loss:

Basic preprocess, no diff-aug

Basic preprocess, diff-aug

Deluxe preprocess, no diff-aug

Deluxe preprocess, diff-aug

If the GAN manages to train, training loss for discriminator and generator should both decrease as time goes on.

Early sample:

Later sample:

The early samples have some hints of the generated image, but there a lot of noise-like colors still in the generated image. In contrast, in the later iterations, the generated image is much closer in resemblance to the reference image, although artifacts are still present.

CycleGAN Results

Results at iteration 1000:

Cat dataset:

PatchDiscriminator, Cycle Consistency:

PatchDiscriminator, No Cycle Consistency:

DCDiscriminator, Cycle Consistency:

Apple dataset:

PatchDiscriminator, Cycle Consistency:

PatchDiscriminator, No Cycle Consistency:

DCDiscriminator, Cycle Consistency:

Discussion of results - cat:

Between using patch and DC discriminator, Patch yields results more similar to the shape of the reference image, but has slightly worse features than the output of DCDiscriminator. As an example, the black & white color of the grumpy cats are shown better with DCDiscriminator, but the face of the cats seems closer to reality with PatchDiscriminator.

Using cycle consistency yields better results, as the one without cycle consistency distorts the output image significantly. However, it does have the color most similar to the target image, as the grey or black & white pattern are all very apparent.

Discussion of results - apple-orange:

Patch vs DC discriminator is a similar story to the previous dataset. DC seems to give a more dramatic color than Patch. In my opinion, patch is much better on this dataset than DC, compared to the cat dataset.

Cycle consistency on and off is also a similar story to the previous dataset. In the case with no cycle consistency loss, there are some results which, although have accurate color, are just blobs of color and do not resemble fruit in any way.