Overview

For the final project, I implemented two subprojects. The first subproject was oriented around poisson blending. The second subproject was based on the lightfield camera and associated work.

Project A

In this project, I implemented poisson blending. The motivation behind poisson blending is to maintain the gradients of a source image without altering the background pixels greatly. The optimization problem is represented as below:

Here, v represents new intensity values, while s represents the source image, and t represents the target image. Indices “i” are taken from the source region, while indices “j” are taken from the corresponding neighboring pixels. To solve the optimization problem, I constructed a sparse matrix A and desired value vector b. Then, I used least squares to find the argmin of the above objective function.

I started off with a “toy problem,” which involved reconstructing an image based on the x and y gradients (and an intensity value taken from the source image). In this toy problem, I only considered two neighbors (the pixel to the right and a pixel above the source pixel). The source image and reconstructed image are shown below:

The maximum error in pixel value between the two images was ~0.001.

Next, I used poisson blending for a more complex problem. Here, I was given a background image, a source image, and a mask. Given these images, I had to appropriately blend the source image onto the background image. Here are the starter images:

For this complex problem, I expanded the neighbor search to one pixel to the right of, to the left of, above, and below the source pixel. The result of the poisson blending was more seamless than the naïve blending approach:

Bells and Whistles (Project A)

For the bells and whistles component of poisson blending, I solved a different optimization problem, which included mixed gradients. The updated optimization problem is as follows:

Here, d_ij corresponds to the mixed gradient. This gradient is the one with a greater magnitude between neighbors in the source image or neighbors in the background image. The result should be a cleaner blend than the poisson blending carried out above:

Project B

In this project, I implemented techniques from work on the light field camera. The data shown here includes rectified images taken from this page. The dataset was collected by taking pictures using a 17x17 grid of cameras.

The first part of this project involved depth refocusing. Here, I parsed the dataset to obtain the u, v shift (u0, v0) for the center camera (8, 8) in the grid. Then, for each image, I applied a x and y-shift (np.roll) to obtain a shifted image. The x-shift was directly proportional to some factor “c” and the difference between the v-shift and v0. Likewise, the y-shift was directly proportional to some factor “c” and the difference between the u-shift and u0. Given all the shifted images, I calculated the average shifted image. This average shifted image would appear to be focused on a certain depth of the scene. I generated 30 different gif frames for 30 different c-values (from -0.25 to 0.25). The result is as follows:

The second part of this project involved Aperture Adjustment. To implement this technique, I started with just the center camera image. Then, I kept increasing the number of images to average over (increasing the search radius) while maintaining a c value of 0.2. This process results in effectively “increasing” the aperture while maintaining focus on the center of the scene. I generated 9 frames (indexed by search radius from 0 to 8). The results are as follows:

Project B Summary

I learned that it is possible to change the focus of a scene by simply averaging photos taken with a camera array. It was especially interesting to see how basic techniques (i.e. shifting individual images and averaging all images) could yield such results.