16726 - Learning Based Image Synthesis - Spring 2020

Tarang Shah (Andrew ID: tarangs)

Homework 2 - Blending images using Gradient Domain Fusion

About this Assignment

This assignment involves blending 2 images. Though the core concept of gradient domain processing can be applied to various image processing tasks, we primarily focus on blending 2 images.

We have a source and a target image. Our goal is to blend the object or texture from the source image to the target image. Naively merging the two images doesn’t produce realistic images. It usually produces an image with abrupt borders between the parts that were merged. We ideally want a smooth blend.

To ensure the images “look” smooth when they are blended, we need to ensure the gradients between the source and the target images are consistent.

Hence we need to find the values of the target pixels for which the gradient of the source region is as close to the original target image. We focus more on the gradient and don’t specifically worry about the pixel intensity here.

We formulate our task as a least squares problem. The task as described below is also known as Poisson Blending.

We can represent our task as

Where,
s - source image
t - target image
v - new intensity values
S - source region
i - a pixel in the source region “S”,
j - each j is a 4 neighbor of “i” (up-down-left-right)

Each summation ensures that the gradient values match those of the source region. In the first summation, the gradient is over two variable pixels; in the second, one pixel is variable and one is in the fixed target region.

We first start with a simple toy problem to ensure the approach works on a single channel toy image and then move on to the full Poisson blending implementation that also includes the ability to handle multi channel images.

Results

Toy Problem

For the toy problem, we can see the same image was reconstructed,

Poisson Blending

Here we have results shown for the provided default image and also for some of my sample images. This includes 2 “good” cases and 1 failure cases as needed.

Original Provided image

Good Image 1

The poisson blending approach essentially works by implementing the equation above. Additionally since the source and target images are different sizes, we need to ensure we are handling that aspect. In this case the assignment code was quite helpful. We took the source image, mask, an offset and constructed a new source image and mask. This new source and mask image are the same size as the target image and have the original source and mask located at the provided offset value. Using the mask, we now generate our A and b matrices that we will use for the least squares solver. Each pixel in general will have 4 equations that we need to handle (top-bottom-left-right). Since the target images are quite large already and have thousands of pixels, storing the values is quite difficult in the RAM. One solution is to use sparse matrices. I used the scipy.sparse functions for this, converting the "A" matrix to a scipy.sparse.lil_matrix was really helpful and also speeded up the execution. Here are a few other images, and also one failure case that I discuss further below

Good Image 2

Not so good Image 1

We can see here that the color of the source image is completely wrong. This is because of a known issue. The issue where we ignored the intensities of the pixels. Since we can see that the cat is red after the blending, it seems that the intensities of the green and blue channels have been reduces a lot to ensure the gradient requirement is fulfilled. This was expected. One way to fix this could be that we explicitly also add some constraint to preserve the relative colors of the image. One way to do this would be to use a different color space instead of the RGB space.

---

CSS Styling and base template based on StackEdit.