This project explores gradient-domain processing, a simple technique with a broad set of applications including blending, tone-mapping, and non-photorealistic rendering. For this project, we will focus on “Poisson blending”; tone-mapping and NPR in the context of image processing.
The goal of this assignment is to seamlessly blend an object or texture from a source image into a target image. The naive method would be to just copy and paste the pixels from one image directly into the other. Unfortunately, this will create very noticeable seams, even if the backgrounds are well-matched. Therefore, we use to Poisson Blending, which is an image processing technique that allows to blend two images seamlessly, utilizing the gradients of both ends, source and target images. A common downside of this algorithm is that the color can change since we are deliberately ignoring the overall intensity. Hence, a red color pixel might turn blue, but we are more interested in the smoothness of the image rather than the texture/intensity.
Figure 0: Movie Generation using Poisson Blending
EXPLANATION
Consider the source image in Figure 1, which has much brighter intensities than the target image. Chart 1, allows us to visualize these intensities as a bar-plot. If we were to paste the pixels f3, f4, f5, f6 into the corresponding pixels in the target image, which is the image we dealt with in the precision section. However, just copying and pasting those pixel values does not give good results as seen in Figure 2 & Chart 2. The source image has much brighter values than the target image. The values of the source image must be modified, in order to make the transition less harsh. To be more specific, some form of blending must be performed between the source and the target image.
Figure 1: Original Image
Figure 3: Histogram of pixel intensities
Another thing that we wish to remark is that even though poisson blending shifts the color of the source image, it still preserves the features of it. In the original source image, f4 is smaller than f3Figure 3: Blended Image
Figure 4: Histogram of Intensities
TOY PROBLEM
To solve poison blending, it might be useful to check the validity of the image reconstruction technique by working on a toy example. In this example, we’ll compute the x and y gradients from an image s, then use all the gradients, plus one-pixel intensity, to reconstruct an image v. Let’s denote the intensity of the source image at (x, y) as s(x, y) and the values of the image to solve for as v(x,y). For each pixel, then, we have three objectives:
Note that the first two constraints could be solved while adding any constant value to v, which might lead to infinite solutions, so we will add one more objective, which forces the color of the first pixels to be the same.
Original.pngReconstructed.pngFigure 5: Results of Toy Problem
POISSON BLENDING
We can use various image processing techniques to blend two images. We will focus on Poisson Blending for this assignment. Consider two images, “source.png” and “target.png”. We would like to blend these two images with the source image being the foreground image and target being the background. To compute the blended image, we will have to create a mask from the source image, align it with respect to the target image, and solve a system of linear equations.
We will be working with these images, and solving a system of equations as given by the following objective function:
As it can be noted that Possion blending works well in majority of the cases, however it is far from perfect. Notice the last row of the results, we try to blend “Bane” from Dark Knight to create the football ground blast scene from the movie. The reason for this failure is that gradient domain smoothing disregards constants, since it is just optimizing over the image gradients. As a result the colors in the synthesized image can come out looking rather strange, like in this case.
RESULTS
Source Target Poisson Blending
BELLS & WHISTLES
MIXED GRADIENTS
This presents an improvement over the Poisson blending algorithm for images where transparency is important. We use the larger magnitude gradient rather than the source gradient.
Below are a few examples that show the advantage of mixed gradients. In the case of text over the wall (see the first row of results), in which we just need to copy the pixels that are in some certain color (the written part of the source image), the mixed gradients work very well. In the case of penguins, walking the boundaries is a bit smoother than the Poisson blending approach. Moreover, when it is needed to blend two facial images, mixed gradients work very well. The mixed gradient method also makes sure that the contrast is preserved as in the case of painting blending (see the sixth row of results).
RESULTS
Naive Mixed Gradient Poisson Blending
COLOR2GRAY
Tone mapping is a technique used in image processing and computer graphics to map one set of colors to another to approximate the appearance of high-dynamic-range images in a medium that has a more limited dynamic range. Tone mapping addresses the problem of strong contrast reduction from the scene radiance to the displayable range while preserving the image details and color appearance important to appreciate the original scene content.
Gradient-domain processing provides one avenue: create a gray image that has similar intensity to the rgb2gray output but has similar contrast to the original RGB image. This is an example of a tone-mapping problem, conceptually similar to that of converting HDR images to RGB displays.
Non-photorealistic rendering (NPR) is an area of computer graphics that focuses on enabling a wide variety of expressive styles for digital art, in contrast to traditional computer graphics, which focuses on photorealism. NPR is inspired by other artistic modes such as painting, drawing, technical illustration, and animated cartoons. NPR has appeared in movies and video games in the form of cel-shaded animation (also known as “toon” shading) as well as in scientific visualization, architectural illustration, and experimental animation. We have used gradients of the three channels of the images to create a new color image.
Figure 6: Clipped outline imagesChannel1.pngChannel2.pngChannel3.pngFigure 7: Clipped channel images after application of gradients
RESULTS
Color Image NPR
Original.pngNPR.pngOriginal.pngNPR.png
VIDEO SYNTHESIS
Using Poission Blending, we have simulated a “short-movie” using a static background image & a moving mask. We have used the background image as the current One World Trade center, and the source image of a fighter plane. We have used the affine transformation learnt in the previous assignment to build a few frames of masks translated in the x coordinate. The fews were then staked to create a movie as shown in the results below. The movie is pretty smooth, and can be a useful application of poission blending as the user is more concered about the overall moive rather than individual frames which may have some artifacts to the naked eye.
