By: Maneesh Bilalpur
This project explores gradient-domain processing, a simple technique with a broad set of applications including blending, tone-mapping, and non-photorealistic rendering. For the core project, we will focus on “Poisson blending”; tone-mapping and NPR can be investigated as bells and whistles. The primary goal of this assignment is to seamlessly blend an object or texture from a source image into a target image. The simplest method would be to just copy and paste the pixels from one image directly into the other. Unfortunately, this will create very noticeable seams, even if the backgrounds are well-matched. How can we get rid of these seams without doing too much perceptual damage to the source region?` Here we take the following approach: The insight we will use is that people often care much more about the gradient of an image than the overall intensity. So we can set up the problem as finding values for the target pixels that maximally preserve the gradient of the source region without changing any of the background pixels. Note that we are making a deliberate decision here to ignore the overall intensity! So a green hat could turn red, but it will still look like a hat. We can formulate our objective as a least squares problem. Given the pixel intensities of the source image “s” and of the target image “t”, we want to solve for new intensity values “v” within the source region “S”: \(\newcommand{\argmin}{arg min}\) \(\boldsymbol{v} = \argmin_{\boldsymbol{v}}\sum_{i \in S, j \in N_i \cap S}((v_i - v_j) - (s_i - s_j))^2 + \sum_{i\in S, j \in N_i \cap \neg S}((v_i - t_j) - (s_i - s_j))^2.\) Here, each “i” is a pixel in the source region “S”, and each “j” is a 4-neighbor of “i”. Each summation guides the gradient values to match those of the source region. In the first summation, the gradient is over two variable pixels; in the second, one pixel is variable and one is in the fixed target region. The method presented above is called “Poisson blending”. Check out the Perez et al. 2003 paper to see sample results, or to wallow in extraneous math. This is just one example of a more general set of gradient-domain processing techniques. The general idea is to create an image by solving for specified pixel intensities and gradients.
In this work we combine two images--a source(or foreground) and target(or background) based so that the gradients are preserved on fusion. A naive blend of copying the intensities to the source image creates artifacts due to potential stark differences in the intensity levels of two images.
The toy problem where we try to synthesize an image such that the source image gradients are preserved throughout returns the original image as shown below.
In Poisson blending we use a mask to highlight what object/stuff we need to incorporate into the target image from the source image. The gradient preservation happens to satisfy two criteria. First, we should ensure that the gradients inside the mask match the gradients in the source image at the same place. Second, we should make sure that the border pixels do not create any artifacts by ensuring that the gradients in the boundary are preserved between the source and target when compared to the source image The following demo example shows it better.
With spongebob we see that images are well blended. Despite the issues with mask along the border(see the naive blend image along spongebob) because the color scheme in the target neighbourhood is similar we do not see any artifacts. We notice that with images where there is a stark difference in color schemes the Poisson blending fails. The sun object essentially became a blue splash next to what looks like a moon in the image. Its is interesting to see that the edge profile is still preserved.
Significant(though not documented) performance boost has been observed by using scipy.sparse module to represent the minimization problem as a MxN matrix and solving the least squares problem using scipy's lstsq when compared to using a numpy based representation or solution.
Overview copied from the assignment website here.