Overview

Sergei Mikhailovich Prokudin-Gorskii (1863-1944) was a pioneering photography who took thousands of color pictures at the time as early as 1907 when color photography was the wave of the future. His idea was simple: record three exposures of every scene onto a glass plate using a red, a green, and a blue filter. Luckily, his RGB glass plate negatives survived and have now been digitized and available online. The goal of this assignment is to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, we need to extract the three color channel images, place them on top of each other, and align them so that they form a single RGB color image. A cool explanation on how the Library of Congress created the color images on their site is available here.

Method

I implemented two types of alignment measure: the Sum of Squared Differences (SSD) distance, and the normalized cross-correlation (NCC). The one that works better (judging by human eyes) will be shown in the following sections. The alignment function is also implemented in two types. The naive one exhaustively searches over a window of possible 2D displacements, which is suiable for small and low-resolution images. A wiser version uses image pyramid technique to downsample the image to different scales (Here I choose a default scaling base of 2), recursively searches over the image pyramid sequentially starting from the coarsest scale (smallest image) and going down the pyramid till the finest scale. This works better on larger high-resolution images and is more efficient.

To further improve the color image quality, I added some Bells & Whistles:

1. Automatic cropping: I use canny edge detection over three color channels, and crop out the rim region that neither channel has any pattern. This is aimed to automatically remove the white and black outer frames of the images. But there are sometimes some manual number markers appeared in the edge making the edge cropping not optimal, which inspired me of #3.
2. Refinement cropping: Remove the edge with weird monocolor or shifted image contents due to np.roll in the aligned results.
3. Recursive refinement: Use the R,G,B channels of the aligned image, reapply the automatic cropping, alignment and refinemnet cropping again.
4. Automatic contrasting: Use histogram equlization to make the color of the reault iamge more realistic.

This is a rough pipeline of the entire process:

1. Extract the three color channel images
2. Automatically crop out black and white edges
3. Adjust scale using the pyramid technique
4. Use blue channel as anchor, align sub-images of red and green images to a blue sub-image
5. Continue alignment on next image pyramid scale
6. Find optimal alignment on the finest scaleand output displacement vectors
7. Stacjthe 3 images according to the displacement vectors
8. Print the reconstructed image

*displacement vectors are shown in format (upright offset, horizontal offset)