Assignment #2 - Gradient Domain Fusion

Background

This project explores gradient-domain processing, a simple technique with a broad set of applications including blending, tone-mapping, and non-photorealistic rendering.

The primary goal of this assignment is to seamlessly blend an object or texture from a source image into a target image. The simplest method would be to just copy and paste the pixels from one image directly into the other. But direct copy-and-paste will not lead to a seamless result.

The approach used in this assignment is to optimize over the image gradients such that we maximally preserve the gradients of the source region without changing any of the pixels in the target image. This means that the color of the source image might change, but it will lead to more consistent gradients around the image seam, thus leading to a perceptually pleasing blend.

We can formulate our objective as a least squares problem. Given the pixel intensities of the source image “s” and of the target image “t”, we want to solve for new intensity values “v” within the source region “S”:

$\newcommand{\argmin}{arg min}$ $\boldsymbol{v} = \argmin_{\boldsymbol{v}}\sum_{i \in S, j \in N_i \cap S}((v_i - v_j) - (s_i - s_j))^2 + \sum_{i\in S, j \in N_i \cap \neg S}((v_i - t_j) - (s_i - s_j))^2.$

Here, each “i” is a pixel in the source region “S”, and each “j” is a 4-neighbor of “i”. Each summation guides the gradient values to match those of the source region. In the first summation, the gradient is over two variable pixels v_i and v_j; in the second, one pixel is variable (v_i) and one is in the fixed target region. The first summation is designed for any pixel that is enclosed within the interior region, meaning that a given 4-neighbor v_j is also within the masked region. The second summation is designed for pixels on the border of the region, meaning that neighbors of the pixel are outside the masked region, and so we try to assign their gradients on the basis of the gradient with respect to the target image pixel t_j.

The method presented above is called “Poisson blending”. Check out the Perez et al. 2003 paper to see sample results. This can also be achieved with a more complex approach using the Laplacian and Divergence of the image field, that is, with the Poisson equations in 2D. However, the least squares optimization approach is simpler to understand.

Part 1.1 Toy Problem

For this toy problem, we use the image gradients along the X and Y axes and reintegrate them (with constraints on the color) to reconstruct the same image again.

If we denote the intensity of the source image at (x, y) as s(x,y) and the values of the image to solve for as v(x,y), then for each pixel, we have two objectives:

1. Minimize $(( v(x+1,y)-v(x,y)) - (s(x+1,y)-s(x,y)) )^2$, so the x-gradients of v should closely match the x-gradients of s.
2. Minimize $(( v(x,y+1)-v(x,y)) - (s(x,y+1)-s(x,y)) )^2$, so the y-gradients of v should closely match the x-gradients of s. Note that these could be solved while adding any constant value to v, so we will add one more objective:

3. minimize $(v(1,1)-s(1,1))^2$

   The top left corners of the two images should be the same color

The resulting input and output is shown below (I added an extra constraint for the bottom pixel too, but otherwise the image still was reconstructed almost perfectly).

Part 1.2 Poisson Blending

Here are some results of Poisson Blending as described above:

Target Image:

Source Image:

Resulting Images (naive and Poisson blended):

This is an interesting timelapse image that I had from field testing drones at Gascola, Penn Hills, earlier this winter. I tried to add in a fighter jet to add to the mania! Although the jet was already in the air, the clouds were of a different color and texture, which led to a very unrealistic naive blend. Using Poisson blending smoothed the gradients and colors over the entire image patch, leading to a very smooth transition between the sky and the jet cutout. Note that the color of the jet also becomes darker as a result, since this method does not guarantee anything about the actual colors.

Additional results of Poisson Blending

Target Image:

Source Image:

Resulting Images (naive and Poisson blended):

Additional results of Poisson Blending

Target Image:

Source Image:

Resulting Images (naive and Poisson blended):

Failure case

Target Image:

Source Image:

Resulting Images (naive and Poisson blended):

Here it can be seen that the seamlessness of the blending has definitely improved, but the gradient values within the image have also caused it to become smoothed/blurred within the masked region. This causes it to look distinctly different from the brick wall that forms the target image. This would look much better with the mixed gradients extra credit.

Bells & Whistles (Extra Points)

Mixed Gradients (10 pts)

Here we can follow the same steps as Poisson blending, but use the gradient in source or target with the larger magnitude as the guide, rather than the source gradient: $\boldsymbol{v} = \argmin_{\boldsymbol{v}} \sum_{i \in S, j \in N_i \cap S} ((v_i - v_j) - d_{ij})^2 + \sum_{i \in S, j \in N_i \cap \neg S}((v_i - t_j) - d_{ij})^2.$ Here “$d_{ij}$” is the value of the gradient from the source or the target image with larger magnitude, i.e.

if abs(s_i - s_j) >= abs (t_i - t_j):
    d_ij = s_i - s_j
else:
    d_ij = t_i - t_j

I implemented this but was unable to get the desired results, even though I am quite sure the implementation is correct.