Assignment #1 - Colorizing The Prokudin-Gorskii Photo Collection
Overview
To approach the project, I started with attempting to minimize the L2 norm over a square window
of displacements (15 pixels). This provided an initial improvement in quality over
the un-aligned images, but the alignment was still significantly off. I then pivoted to the normalized cross-correlation
approach, but found strangely that this resulted in an even worse alignment than using the L2 norm.
Reading through the "Final Advice" section, I realized that it was possible that the image borders were
causing an issue with the alignment. I then implemented a function to slice off the borders prior to computing
the alignment score, but this did not seem to help either. Eventually I went to office hours, and together with Viraj realized that
there was likely an issue with how I was cutting off the borders - I was cutting off the borders after the inital call to np.roll,
so cutting off the borders at that point would have left the actual border intact and cut off valuable parts of the images.
Even after fixing this issue, the alignment still appeared to be incorrect.
Another suggestion I had gotten at office hours was to print the displacement vector output from my align funciton,
and then try manually adjusting the displacement in order to find the correct displacement vector (this was all done using the cathedral
example image). After doing this, I found that the correct displacement was actually the output of my alignment method (with normalized cross-correlation)
with the x-y coordinates swapped! (turned out that my final call to np.roll passed in the displacement vector in the wrong order).
Upon fixing this issue, the cathedral image aligned correctly. I then worked on implementing the coarse-to-fine pyramid to align the larger .tiff images more efficiently.
My approach for this was to scale the image down using the skimage.transform rescale function, and make recursive calls to my original implementation.
At the coarsest scale, I searched the full 15 pixel radius, and then at finer scales I searched a smaller radius, as the alignment
was already roughly estimated from the search at the coarser-scale. To refine the alignment, I mainly focused on adjusting the parameters
for how much of the border to cut off, the search-window, and the pyramid height.
Results
Green displacement: (2,5)
Red displacement: (3,12)
Green displacement: (14,42)
Red displacement: (26,95)
Green displacement: (10,59)
Red displacement: (7,119)
Green displacement: (16,41)
Red displacement: (22,90)
Green displacement: (-6,57)
Red displacement: (-14,101)
Green displacement: (10,76)
Red displacement: (-5,119)
Green displacement: (5,52)
Red displacement: (7,110)
Green displacement: (-3,40)
Red displacement: (18,91)
Green displacement: (4,57)
Red displacement: (10,108)
Green displacement: (-3,40)
Red displacement: (18,91)
Images of Choosing
From the collection, I selected two images to test the alignment algorithm on - the translated title of the first work being
"City of Lodeinoe Pole. Monument to Emperor Peter the Great", and the second "Signal tower and storehouses of the M.P.S. [Ministry of Communication and Transportation] in Suvaldy"
Green displacement: (10,21)
Red displacement: (10,50)
Green displacement: (2,19)
Red displacement: (2,49)
Alignment Issues
The resulting algorithm was largely sucessful on most the images depicted above. The only image
out of the ones experimented on for the first project that the algorithm failed to align was the file
titled "self_portrait.tif", and additionally it struggled to align "lady.tif". For the former image, I hypothesize
that this resulted from using the pixel brightness values for evaluating similarity. The dark water
and numerous scattered jagged rocks posed a challenge for the alignment, and using edges or gradient detection to measure
similarity would likely solve this issue.
For "lady.tif", the resulting alignment upon zooming in appears a couple of pixels off in the y-direction.
This image is dark for the most part, so it is likely that rescaling the image intensities such that the darkest
pixel is 0 and the lightest pixel is 1 as suggested in the bells and whistles section would solve this problem.
Extra Credit
For the extra credit, I attempted the "Automatic contrasting" task, leveraging the skimage.exposure module.
To perform this auto-contrasting, the channels are rescaled such that the brightest pixel on the brightest
channel is 1, and the darkest pixel on the darkest channel is 0. To visually analyze the difference,
I applied this to the momument photo that I pulled from the collection. The difference is subtle due to the fact
that the pixels intensities are already close to the 0 min and 1 max (although not quite), but some results that I noticed were
a pop in the color of the fencing in front of the monument, as well as the whiteness of the columns.
Here is what the image looks like before applying the contrast:
And here is the resulting image after the contrast:
