Overview

This project explores gradient-domain processing, a simple technique with a broad set of applications including blending, tone-mapping, and non-photorealistic rendering. The primary goal of this assignment is to seamlessly blend an object or texture from a source image into a target image.

Approach

First, we'll discuss the approach for the toy problem. This was a simplified version of gradient domain processing where the high-level goal was to reconstruct an image. This was achieved by first computing the x and y gradients from the input image as well as the intensity of one pixel, and then solving a least squares problem where we try to minimize the differences in the x and y gradients of our output image relative to our input. We also introduce a singular pixel intensity constraint to control for color. The system is solved efficiently using sparse matrices and a least squares solver.

For the more general problem, the high-level goal is to copy paste a specified source region (from a source image) into a target image. While this can be done naively by directly copying the pixels, this will almost always lead to bad results unless the backgrounds or images are from a similar distribution. Thus, we adopt a similar approach to the toy problem based on gradients where we solve a least squares problem following two types of constraints. For the first type, for any two neighboring pixels in the specified source region, we minimize the difference in gradients between these and the corresponding pixels in our output image. For pixels outside of the source region, we directly copy the pixels from the target image. This motivates the second type of constraint: for any two neighboring pixels where one is in the source region and one is not, we minimize the difference between the corresponding gradient in the source image and the intensity difference of the output pixel in the source region with the target pixel not in the region. Again, we can solve this relatively efficiently using sparse matrices and a least squares solver for medium resolution images.

Finally, we also implement the Bells & Whistles assignment of mixed gradients. For this approach, instead of always minimizing the least squares error against the gradient of the neighboring pixels in the source image, we minimize against the gradient from either the source or target image depending on which has larger magnitude.

Toy Problem and Demo Data Results

First, we present results on the toy problem and example images provided as part of the assignment. Below, we can see the results on the toy problem.

  • The input grayscale image.

  • The reconstructed output.

While there is some slight bluriness in the reconstruction due to resizing for display on this website, it does a pretty good job overall. We can also see how poisson blending does on the demo data of inserting a bear into a pool.

  • The source image of the bear.

  • The target image of the pool.

  • Naively pasting the pixels directly

  • Using poisson blending

While the color of the bear's fur changes, there's a clear difference in quality between using poisson blending and naively pasting pixels!

Poisson Blending Results

Now, we present results on custom images. First, my favorite blending result is inserting a picture of my cat (Mimi) into a picture of my friend's room.

  • The source image of Mimi

  • The target image of my friend Joey in his room

  • Naively pasting the pixels directly

  • Using poisson blending

Unfortunately, Mimi turns an orange-ish color to match the floor, but the final result is still an improvement over the naive method, especially around the edges. Let's see an example where we try to insert the Death Star above Gates.

  • The source image of the Death Star

  • The target image of Gates at CMU

  • Naively pasting the pixels directly

  • Using poisson blending

Very ominous! Finally, we show an example of a dolphin jumping out of a volcano.

  • The source image of the dolphin

  • The target image of the volcano

  • Naively pasting the pixels directly

  • Using poisson blending

Poisson Blending Failure Cases and Mixed Gradients Results

Normal poisson blending generally works pretty well, but it can have trouble when the source region has a drastically different background from the target image. Using mixed gradients, we can solve these failure cases by choosing our output pixel values based on the gradients with largest magnitude instead of always using the source gradients. In the example below, there's a clear difference between the two methods when we try to paste a picture of my handwritten name onto a brick wall.

  • The source image of my handwritten name

  • The target image of a brick wall covered with graffiti

  • Naively pasting the pixels directly

  • Using poisson blending

  • Using mixed gradients

With vanilla poisson blending, the white background behind my name becomes a muddled yellow in the output picture. Using mixed gradients produces an image which replicates the more dominant gradients of the target image. Finally, we can look at an example which demonstrates the more subtle differences between poisson blending and mixed gradients.

  • The source image of the dolphin

  • The target image of the volcano

  • Naively pasting the pixels directly

  • Using poisson blending

  • Using mixed gradients

It's clear that both poisson blending and mixed gradients are better than directly copying pixels. While the difference between the two is subtle, mixed gradients preserves more of the volcanic eruption around the cone and illuminates the dolphin's tail accordingly.