16-726: Assignment #2 - Gradient Domain Fusion

Rawal Khirodkar

1. Brief Description: In this project, we want to seamlessly blend an object or texture from a source image into a target image using gradient domain processing. The key idea is to use Poisson Blending - we compute the values of the target pixels such that they are similar to the gradient in the source image in the masked area without changing the background pixels. As result, the color of the image can possibly change but the texture and shape will not. This done by solving a least square objective highlighted in the handout. The implementation uses some tricks such as scaling the image intensities between [0, 1] and using sparse matrices for memory efficiency.

2. Toy Problem: Here is the reconstructed image for the toy problem by solving the least square objective. I used a border of 1 pixel around the source image for handling the neighbour out of image case.

3. Poisson Blending: Here are the results. I used scipy.sparse.lil_matrix to construct the A matrix but then switched to csr_matrix for solving the lsqr problem. Also, I have added the equation which appear multiple times, as a result my matrix A is of dimension num_variables X num_variables and b is of dimension num_variables X num_channels. After solving the lsqr objective, I found it important to clip the solution in range [0, 1], the extreme values detroited the quality of the image.

4. Failure cases: This was fun! I tried inserting Ironman in a photo from Wanda Vision (please do watch it!). The color changes makes the ironman look weird on the road, infact the naive blend looks better as the mask is almost pixel accurate. I also tried inserting myself in my girlfriend's photo, the texture of my background is significantly different from the target photo which makes the edges of the blending look artificial.

5. Mixed Gradients: Straightforward implementation except the "if" conditions checking the magnitude of the gradient between source and target. Here are some results

6. Color2Gray: I converted the image to HSV space and only used the gradients in S and V channel for running mixed gradients, as the Hue channel only represents the color. This enables us to preserve the image contrast.