CMU 16726
Project 1 - Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection

Harsh Sharma

Problem Background







Sergei Mikhailovich Prokudin-Gorskii was able to capture color images by taking three exposures of the same scene, through red, green, and blue filters. In this project, I created an algorithm to create a color image from the three different exposures by cropping the images corresponding to different channels and then aligning, and finally stacking them. The method of evaluation was the sum of squared differences/normalized cross correlation between pixels, and for larger images, a recursive image pyramid algorithm was applied to reduce the search window and speed up processing time .





Cathedral

Single Scale Method

Using SSD and NCC as evaluation metric on the pixels, and keeping the blue channel as reference, this method aligned red and green channels so as to minimize the loss. The results were surprisingly very good.

CathedralCathedral: No Shift

CathedralCathedral: Aligned using SSD Red[3, 12], Green[2, 5]

CathedralCathedral: Aligned using NCC Red[3, 12], Green[2, 5]


































Image Pyramids

For larger files like the tif's, computing via searching over a window could be very time consuming. Downsampling the image helps to get an estimate of the perfect alignment and the image is then upsampled using the initialization from the previously downsampled image. Images were shrunk by 50% recursively until the minimum dimension (in pixels) was less than 400. At every resolution single scale processing was done over the [-15, 15] window. Finally, we would rescale up while running SSD on a [-15, 15] displacement window to further refine the final displacement values. This sped the processing time up significantly as we only scan the 900 pixel window once, at the base case of recursion and on the smallest image.

Throughout features were considered to be pixels. Some images could have been better aligned in particular, Emir had I tried with gradients or some other feature from the image. Using pixels as the features is very prone to errors while matching exposures as it is possible that an image may have had different brightness levels for different channels. Results are shown below.
















LadyLady Unaligned

MelonsHarvesters Unaligned

Onion ChurchSelf Portrait unaligned




























LadyLady SSD: Red[-4, 136], Green[0, 55]

HarvestersHarvesters SSD: Red[12, 178], Green[4, 82]

Self PortraitSelf Portrait aligned SSD: Red[36,108], Green[25, 52]




























Self PortraitLady aligned using NCC Red: [11,116]. Green : [8,54]

Harvesters Aligned using NCC Aligned using NCC Red: [14,123]. Green : [17, 59]

TrainAligned using NCC [14,123]. Green : [17, 59]





three_generationsUnaligned

three_generationsAligned using SSD

three_generationsAligned using NCC Red:[12, 110],Green:[14, 52]

EmirEmir Unaligned

Emir_alAligned using SSD

Emir_alAligned using NCC Red:[39, 70],Green:[24, 48]

EmirVillage Unaligned

Emir_alVillage aligned using SSD

Emir_alVillage aligned using NCC Red:[22, 137],Green:[12, 64]

EmirTurkmen Unaligned

Emir_alTurkmen aligned using SSD Red:[28, 116],Green:[21, 55]

Emir_alTurkmen aligned using NCC Red:[28, 116],Green:[21, 55]

EmirTrain Unaligned

Emir_alTrain aligned using SSD Red:[28, 116],Green:[21, 55]

Emir_alTrain aligned using NCC Red:[32, 86],Green:[6, 42]

IconIcon Unaligned

IconIcon aligned using SSD Red:[28, 116],Green:[21, 55]

IconIcon aligned using NCC Red:[23, 89],Green:[17, 41]