Final Project

16-726 LEARNING-BASED IMAGE SYNTHESIS

Create an practical and interactive web application by using a synthesis techniques to showcase to normal users

Texture synthesis have many applications in gaming and computer graphics, and one practical application that everyone can relate to is for clothing.

User can upload their own texture images
Could not use pretrained model from python. The model has to be trained after the user upload their image.
The synthesis has to be done real-time so that the users can see the results.
10 hours of training is perfectly fine for a research paper, but normal users probably go away after 10 seconds of waiting.
In addition, users should be able to change color of the textures to give them more creative power

The most critical requirement is time performance as the application needs to be run in real-time.
GAN training is difficult to curate to just one user input image.
GAN training requires a significant amount of time to train such as in this paper.
Therefore, a reasonably good approach for texture synthesis is to use style transfer to generate the texture.
Use MobileNetV2 to extract style loss instead of vgg19 for faster performance

User can choose one color for the clothes texture.
Using color transfer approach by using some type of color loss is overkill and slow. Changing the color after the texture synthesis training is better in performance, and user can try different colors without retraining the model.
As a result, I simply shifted the mean of the RGB channels toward the user-selected color to have the color of the texture match the color

As the key requirement for this project is allow users to run everything on client-side, so all of the project were coded in Javascript with the following technology stacks:
Tensorflow.JS for texture synthesis training on client side
OpenCV.JS for color manipulation
Angular framework building the application

Using Javascripts to replicate python code is not straightforward. Javascripts cannot access torch/tensor index easily and broadcasting like in pytorch or numpy. Methods in Tensorflow.js is limited compared to tensorflow in python
Training operations take quite some time so I needed to code a way so that it doesn’t block the responsiveness of the UI
Explicit memory management is required in Tensorflow.JS to avoid memory leak in the browser as the training takes more iterations.
The application might not be able to train on some mobile devices due to computation constraints.