My Self Introduction

Colorizing the Prokudin-Gorskii Photo Collection

This project is an attempt to formulate and test an approach to aligning the blue, red and green channels of the Prokudin-Gorskii photo collection.

For smaller images, an exhaustive search was conducted on various translation offsets for each channel. Each pair of translation offsets we analyzed for similarity using an L2 error on an inner subset of pixels.
For larger images, a Laplacian pyramid was computed and image alignment was performed at each step.

The resultant images are rescaled for size and shown below, together with the calculated offset for alignment for each channel.

Channel Alignment for Example Images

Each image has been resized to a height of 1080px while maintaining aspect ratio.

Image Title G: [row, col] | R: [row, col]

The offsets (row, col) for each channel (G, R) with respect to B can be found from hovering over the image.

Cathedral G: [5, 2] | R: [12, 3]

Emir G: [49, 24] | R: [103, 43]

Harvesters G: [60, 16] | R: [124, 14]

Icon G: [41, 17] | R: [89, 23]

Lady G: [55, 8] | R: [110, 12]

Self Portrait G: [79, 29] | R: [175, 34]

Three Generations G: [55, 13] | R: [112, 10]

Train G: [42, 6] | R: [87, 32]

Turkmen G: [56, 20] | R: [114, 27]

Village G: [65, 12] | R: [138, 23]

Challenges

While looping over all possible translations was relatively straightforward, tuning the scale factor for the Laplacian pyramids was a challenge.
If the factor was too small, the offsets calculated would vary too much, and could sometimes be translated too far off.
If the factor was too large, the runtime would suffer as a larger search space would have to be used with images of a higher resolution.
Initially, I also did not take into account the edge effects, leading to poor alignment. This issue was solved after removing a fraction of each channel's edge.

Channel Alignment for Extra Images from the Photo Collection

Example 1 G: [101, -5] | R: [-5, -5]

Example 2 G: [27, -4] | R: [62, -17]

Example 3 G: [39, 2] | R: [88, 8]

Example 4 G: [49, 19] | R: [100, 28]

Example 5 G: [54, 11] | R: [117, 17]

Example 6 G: [33, 20] | R: [80, 44]

Bells and Whistles

Using Gradient Features for Alignment

Instead of computing the loss over normalized pixel values, it might be advantageous to extract edge features from each channel and match those instead.

Using an xy-Sobel filter, edge features from each channel are extracted.

Then, a similar procedure as above is used for channel matching.

In comparing both sets of images, it is relatively difficult to see any differences since the original alignment works relatively well.

Cathedral G: [5, 2] | R: [12, 3]

Emir G: [49, 24] | R: [111, 48]

Harvesters G: [56, 12] | R: [123, 14]

Icon G: [39, 16] | R: [88, 23]

Lady G: [57, 9] | R: [120, 13]

Self Portrait G: [74, 25] | R: [175, 37]

Three Generations G: [59, 16] | R: [115, 12]

Train G: [53, 5] | R: [84, 28]

Turkmen G: [58, 21] | R: [117, 29]

Village G: [65, 11] | R: [138, 22]

Edge Detection

Edge detection was formulated as a two-step process.

Firstly, given that the green and red channels were rolled, any portion of the image that was rolled over was removed.

Secondly, a horizontal and vertical Sobel filter was run across the three channels of the image.

Then, depending on the orientation of the filter, a mean value for the pixels of each column / row was computed.

Since higher values correspond to a higher chance of a straight edge, given a threshold value, the last spike value was calculated for each of the four edges and three channels.

All pixels that were to the side of these extreme spikes were then removed.

The results on several examples are shown below.