16-726 Learning Based Image Synthesis

Colorizing the Prokudin-Gorskii photo collection

By: Sanil Pande

Introduction

This color photo collection is of separate red, green, and blue color channels on high resolution images.

The challenge is to develop an algorithm that both runs fast and is able to find the alignment between all three channels correctly..

Approach

Naive Implementation

Since the channels are misaligned, simple stacking the three channels on top of each other does not work.

The next immediately obvious method is to exhaustively search over all alignment pixels. This works well for smaller images, but at a higher resolution of greater than 1024 pixes, the method becomes prohibitively slow.

Image Pyramid Alignment

A smarter method is to align the the channels exhaustively at smaller scales (~128 pixels), and then use these results to fine-tune the search over larger scales.

This was implemented here using a recursive algorithm using a window size of 5 pixels on each side, which works quite fast and under one minute for each image.

Using Edges

Even after using the pyramid method, there are some problems. Specifically, normalized cross correlation on the raw channels does not work well due to changes in intensity.

To remedy this, I used a Sobel filter, and summed the absolute values of the responses along both the vertical and horizontal axes.

Using these responses to calculate the NCC for a particular offset gives excellent results, and all the given images are aligned correctly.

Final Results

Here are the final results of the colorization algorithm on the test set, along with the displacements of each non-base filter. The small .jpg images took an average of around 1 second to run, while the .tif images finished in about half a minute each.