16-726 21spring Assignment #2

Gradient Domain Fusion

Author: Zhe Huang (zhehuang)

---

1. Introduction

In this assignment, we explored a gradient-domain processing image blending technique called poisson blending. By minimizing the difference in gradients, we can blend two images together with minimal style discrepancies.

Specifically, we define the image containing interesting objects that we want to put on top of the other image as foreground image, whereas the other image providing the background as background image. Poisson blending is to find a new blended image where each pixel from that image is from the least-square approximation of a solution of a linear system that minimizes the gradient difference between foreground image and the blended image in foreground area and minimizes the gradient difference between background image and the blended image in background area of blended image.

from main import *

%matplotlib notebook

2. Toy Problem

In this part, we show the result corresponding to Part 1.1 in the assignment by properly implementing the three constraints stated in the Part 1.1 description. Our output image is basically the same as the imput (with the R1 loss to be less than 0.006), which indicates the correctness of my implementation.

image = imageio.imread('./demo_data/toy_problem.png')
image_hat = toy_recon(image)

plt.subplot(121)
plt.imshow(image)
plt.title('Input')
plt.subplot(122)
plt.imshow(image_hat)
plt.title('Output')
plt.show()

(array([88.00024694, 88.00037218, 88.00043603, ..., 87.99999764,
       87.9999933 , 87.99999397]), 1, 425, 0.0055813396134444734, 0.0055813396134444734, 41.22216598530951, 2955.0834239453943, 0.0026736155084947015, 13768.639281986985, array([0., 0., 0., ..., 0., 0., 0.]))

2. Poisson Blending

In this part, we show the result corresponding to Part 1.2 in the assignment by properly implementing the original Poisson Blending algorithm stated in the Part 1.2 and Background part of the project. The interface for plotting the graph is as follows.

def plt_blend(fg_path, bg_path, mask_path, ratio=1, offset=None, **kwargs):
    fg = cv2.resize(imageio.imread(fg_path), (0, 0), fx=ratio, fy=ratio)
    bg = cv2.resize(imageio.imread(bg_path), (0, 0), fx=ratio, fy=ratio)
    mask = cv2.resize(imageio.imread(mask_path), (0, 0), fx=ratio, fy=ratio)
    if offset is not None:
        fg, mask = align_source(fg, bg, mask, offset)
    fg = fg / fg.max()
    bg = bg / bg.max()
    mask = (mask > 0)
    blend_img = poisson_blend(fg, mask, bg, **kwargs)

    plt.subplot(141)
    plt.imshow(fg)
    plt.title('Foreground')
    plt.subplot(142)
    plt.imshow(bg)
    plt.title('Background')
    plt.subplot(143)
    plt.imshow(fg * mask + bg * (1 - mask))
    plt.title('Naive Blend')
    plt.subplot(144)
    plt.imshow(blend_img)
    plt.title('Poisson Blend')
    plt.show()

2.1 Good Results

Here are two good results of what we get from the Poisson blending.

plt_blend('./demo_data/source_01.jpg', './demo_data/target_01.jpg', './demo_data/mask_01.jpg', 
          offset=-np.array([5, 200]))

(array([0., 0., 0., ..., 0., 0., 0.]), 2, 282, 2.4990412850463346, 2.4990412850463346, 47.50054342763299, 1255.7499858344274, 1.1405100070021637e-06, 33.303433430882166, array([0., 0., 0., ..., 0., 0., 0.]))
(array([0., 0., 0., ..., 0., 0., 0.]), 2, 285, 2.4170406176842487, 2.4170406176842487, 47.74476996395195, 1266.2933426180944, 1.0802273432434826e-06, 43.40826408030195, array([0., 0., 0., ..., 0., 0., 0.]))
(array([0., 0., 0., ..., 0., 0., 0.]), 2, 286, 2.387191026580099, 2.387191026580099, 47.82773113175209, 1269.6794695616347, 1.0650571663636474e-06, 48.593905807378206, array([0., 0., 0., ..., 0., 0., 0.]))

plt_blend('./demo_data/penguin_fg_newsource.png', './demo_data/penguin_bg.jpeg', './demo_data/penguin_bg_mask.png')

(array([0., 0., 0., ..., 0., 0., 0.]), 2, 299, 0.8167625934449271, 0.8167625934449271, 48.904353371546094, 1332.1790504691155, 3.9645974631863275e-07, 54.41712184269822, array([0., 0., 0., ..., 0., 0., 0.]))
(array([0., 0., 0., ..., 0., 0., 0.]), 2, 299, 0.8103262733439978, 0.8103262733439978, 48.90354103187926, 1331.3340997469413, 3.836388438609942e-07, 55.85802130206648, array([0., 0., 0., ..., 0., 0., 0.]))
(array([0., 0., 0., ..., 0., 0., 0.]), 2, 298, 0.8152408931675936, 0.8152408931675936, 48.821816356614484, 1326.0967722766281, 3.973213111709879e-07, 57.236342065157615, array([0., 0., 0., ..., 0., 0., 0.]))

2.2 Bad Results

Here is some not very successful demostration of Poisson Blending. I was meant to merge this grin face and the doge face together but what I got is just the grin face on top of the doge with a different color vairation. The reason why it failed is probably due to that this problem is not suitable for Poisson Blending technique. While Poisson Blending is succesful at blending an object of interest into its surrounding background of a target image, it cannot naturally blend two object together by solving a simple least-square system.

plt_blend('./demo_data/grin2_fg_newsource.png', './demo_data/doge2_bg.jpeg', './demo_data/doge2_bg_mask.png')

(array([0., 0., 0., ..., 0., 0., 0.]), 2, 269, 1.061957365701435, 1.061957365701435, 46.362707029216224, 1198.458204222535, 4.725114853654022e-07, 56.07860017054969, array([0., 0., 0., ..., 0., 0., 0.]))
(array([0., 0., 0., ..., 0., 0., 0.]), 2, 266, 1.1942103347819464, 1.1942103347819464, 46.102856086232315, 1185.7975522018946, 5.450341673254936e-07, 40.10012449844007, array([0., 0., 0., ..., 0., 0., 0.]))
(array([0., 0., 0., ..., 0., 0., 0.]), 2, 272, 1.3801749776784955, 1.3801749776784955, 46.61802116641138, 1214.1458527442267, 6.328928006418275e-07, 24.854696871356378, array([0., 0., 0., ..., 0., 0., 0.]))

3 Bells & Whistles: Mixed Gradients

In this part, we show the effectiveness of using mixed gradients in Poisson Blending. We show the comparison between the original Poisson Blending and the Poisson Blending with mixed gradients, which has a great improvement to seamlessly blend the foreground and background together.

3.1 Original Poisson Blending

In this case, you can clearly see the artifact around the sigma letter in the final blending, which resembles some kind of Gaussian blur.

plt_blend('./demo_data/sigma_fg_newsource.png', './demo_data/brick_bg.jpeg', './demo_data/brick_bg_mask.png')

(array([0., 0., 0., ..., 0., 0., 0.]), 2, 270, 3.3969393201023155, 3.3969393201023155, 46.47729574226819, 1104.616975756326, 1.5581081362271105e-06, 61.34624074877364, array([0., 0., 0., ..., 0., 0., 0.]))
(array([0., 0., 0., ..., 0., 0., 0.]), 2, 278, 4.557584819086786, 4.557584819086786, 47.162818612550765, 1173.8457040748306, 2.1143913228916857e-06, 59.12084206423772, array([0., 0., 0., ..., 0., 0., 0.]))
(array([0., 0., 0., ..., 0., 0., 0.]), 2, 279, 4.399486073446476, 4.399486073446476, 47.24796459365762, 1178.7845983955347, 1.9678607632302844e-06, 59.80695304665789, array([0., 0., 0., ..., 0., 0., 0.]))

3.2 Mixed Gradient Poisson Blending

In this case, you can clearly see the improvement from the origianl Poisson Blending, where in the final product, the sigma letter is seamlessly blended into the bricks, without blurry artifacts.

plt_blend('./demo_data/sigma_fg_newsource.png', './demo_data/brick_bg.jpeg', './demo_data/brick_bg_mask.png',
    d_ij=mixed_grad)

(array([0., 0., 0., ..., 0., 0., 0.]), 2, 266, 4.278848368033333, 4.278848368033333, 46.13113078643271, 1087.0198793177885, 1.9414231465469504e-06, 63.0358454955352, array([0., 0., 0., ..., 0., 0., 0.]))
(array([0., 0., 0., ..., 0., 0., 0.]), 2, 268, 5.932915604659525, 5.932915604659525, 46.30565202192158, 1128.3913915277872, 2.612787366949068e-06, 56.64837141768427, array([0., 0., 0., ..., 0., 0., 0.]))
(array([0., 0., 0., ..., 0., 0., 0.]), 2, 268, 5.7001968522457895, 5.7001968522457895, 46.305483640539656, 1128.913725566812, 2.602517891802814e-06, 58.15660952377921, array([0., 0., 0., ..., 0., 0., 0.]))