Assignment #2 - Gradient Domain Fusion

Tarasha Khurana (Andrew ID: tkhurana)

Overview

This homework aims to explore the concept of gradient domain fusion, primarily for the task of blending. We work with Poisson Blending in order to seamlessly blend an object or texture from a source image into a target image while preserving gradients (but not necessarily pixel intensity). For the task of Poisson Blending, we aim to minimize two 2-norms; gradients in the source image should match the gradients in the target image inside the blending region, and the gradients on the boundary of the blending region should match the gradients of the removed region from the target image. Minimizing both these constraints blends the source region into the target image.

Toy Problem

The toy problem gave us a taste of poisson blending in an easy case - where the mask/blending region is the entire image. We solved for the two minimization problems while keeping the top left pixel value constant as a constraint for the minimizations. Solving this with least squares recovered the original image.

Poisson Blending

For this part of the assignment, I relied on the masking_code.py script for getting the source image mask and an aligned source image to be pasted onto the desired target image region. Once I had these, I only had to code up the equations that would fill up the A and b matrices (similar to the toy problem).

For the source pixels that had their neighbours inside the source region, the equations were straightforward. However for the ones that were not inside the source region, I had to find and store the locations of the border pixels of the source region. I did this by once going from top to bottom in the mask image and checking for change of gradients and once, going left to right. I additionally constrained the optimization by fixing the intensity of the border pixels to the corresponding intensity in the target image.

I used scipy.sparse.lil_matrix and scipy.sparse.linalg.lsqr to create a sparse A matrix and do a least squares minimization on this sparse matrix. This reduced the computation time and I could process pretty big images quickly. Some examples of poisson blending are shown below.

Example 1

Example 2

Example 3

The following failure case arises when not background of the object in the source image is not the same as the background of the target image. This shows that the source and target regions should have similar backgrounds or the poisson blending does not work seamlessly.

Example 4

Bells and Whistles: Mixed Gradients

I also implemented the Mixed Gradients part of assignment as my code was written in a way that it was only a few lines of change. Mostly I found that using mixed gradients from the source and the target image, resulted in a better blending result that took care of the target region edges/texture at the place where the source images was to be pasted. Some comparisons between poisson blending and mixed gradients is shown below.

Example 1

Example 2