16-726 Learning-Based Image Synthesis

Project 2: Gradient Domain Fusion

Trung Nguyen

Objective

The goal of this assignment is to seamlessly blend one image to another target image with gradient-domain processing. Specifically, we focus on Poisson blending to achieve the task.

Preparation

The most naive way to do the task without any seamlessness is simply copying and pasting pixels from one image to another. However, such an approach would lead to obvious seams on the image. We try to remove the seam such that the difference between the boundary id minimal. The approach could be described as below

Preparation: To do the blending, we need a mask image to specify which section in the source to blend in another image. The way to do that is to manually draw a polygon on the source image to use for the blending. The sample masking is similar to below

Approach

After preparing the mask image, we are given one source image, one target image, and one mask image to specify which region of the source image to put on the target image. At a high level, we will try to put a source image to the target image so that the boundary maintains the gradient of the source image. This way, we can ensure that the source image won't have different lighting and contrast than the target image.

As a result, we form a least-square problem of constructing the pixels in the target image so that these pixels satisfy the following properties:

The constructed pixels gradients in both x and y directions must be close to the gradients in the source image so that the content of the image is preserved.
For each pixel x,y, inside the reconstructed region, we have a constraint that
v(x+1, y) - v(x,y) = s(x+1,y) - s(x, y)
v(x, y+1) - v(x,y) = s(x,y +1) - s(x, y)
The non-mask pixels must be close to the target images
For each pixel u, v at the boundary, we have a constraint that
v(u, v) = s(u, v)

With these constraints, we can form a combination of equations of

Av = b

where A is a sparse matrix with each column correspond to one pixel in the region
v is the vectorized vector of the image region to reconstructed
b is the right-hand side of the constraints

Therefore, we would want to minimize

Av - b

with least-square optimization. Because A is a huge and sparse matrix, we can use scipy's sparse Matrix to construct A to optimize the performance and the memory footprint. As a result, the speed to run the toy problem is instant.

Toy problem

Before solving the original problem, we will solve a toy problem of reconstructing an image using the same gradient constraints to validate the approach.

The constructed pixels gradients in both x and y directions must be the same to the gradients in the source image so that the content of the image is preserved.
For each pixel x,y, of the image, we have a constraint that
v(x+1, y) - v(x,y) = s(x+1,y) - s(x, y)
v(x, y+1) - v(x,y) = s(x,y +1) - s(x, y)
The top left pixel must be the same as the original image
v(0, 0) = s(0, 0)

Poisson blending results

This is my most favourite blend because the white background of the snow blended really well with the snow background in the source, making the result more natural.

These two images are not very well blended as the above pair because of the difference in the colour of the snow at the bottom of the penguin and the tree at the top of the penguin that makes it difficult to match the gradient

Putting text of white background on a target image doesn't work very well at all because the plain background has a gradient of zeros.

Bell and Whistles

Mixed Gradients

A simple trick to handle the above case is simply choosing the gradient with more intensity from both the source and target image. Making the blend look much better for plain background source images.

Color2Gray

Gradient-domain data is also useful in converting colour images by making mixed gradient constraints on channels (such as channel Saturation and Value in HSV or even channel Saturation with the regular converted grey image to restore the intensity in saturation)