Assignment 3

NAME: Hiresh Gupta

ANDREW ID: hireshg

Late days used: 0

1. Differentiable Volume Rendering

1.3. Ray sampling (10 points)

Usage:

python main.py --config-name=box

Outputs of grid/ray visualization:

1.4. Point sampling (10 points)

I have implemented StratifiedSampler in sampler.py for Ray Sampling.

Usage:

python main.py --config-name=box

Point Samples output:

1.5. Volume rendering (30 points)

Implement volume rendering for an SDF volume.

Usage:

python main.py --config-name=box

Color & Depth Visualizations:

2. Optimizing a basic implicit volume

2.1. Random ray sampling (5 points)

I have implemented get_random_pixels_from_image in ray_utils.py for this.

2.2. Loss and training (5 points)

Usage:

python main.py --config-name=train_box

The optimized center and side lengths of the box after training can be found below:

Box centers: (0.2507, 0.2495, 0.0053)
Box side lengths: (1.999, 1.4942, 1.5404)

2.3. Visualization

The rendered volume visualization can be found below:

3. Optimizing a Neural Radiance Field (NeRF) (30 points)

I have implemented the NeRF MLP (without view dependence). The model can be trained by the following command:

python main.py --config-name=nerf_lego

Model output Visualization:

4. NeRF Extras (Choose at least one! More than one is extra credit)

4.1 View Dependence (10 pts)

I have added view dependence to my previous NeRF model. The model can be trained by the following command:

python main.py --config-name=nerf_lego_view_dependence

Model output Visualization:

Discussion:

As seen above, we can see that the model outputs get slightly better on adding view dependence. Without handling view dependence, the previous NeRF model (in Q3) solely predicts the color & density value based on the 3D location. Hence we see some weird artifacts at the back of the lego in Q3 output.

On adding view dependence, we observe a much more realistic output that changes with the viewing direction. This aligns with the observation shared by the authors in their paper that view dependence helps in recreating the specular reflection on the bulldozer tread.

4.3 High Resolution Imagery (10 pts)

Based on the viewing direction results observed earlier, we had established that adding view dependence helps in generating realistic outputs. So I have used it in my other experiments on running NeRF model on higher resolution images. We can run the NeRF model on high resolution imagery by running the following command:

python main.py --config-name=nerf_lego_highres

Additionally, I also played around the NeRF MLP architecture. The command to train a lighter NeRF model can be found below:

python main.py --config-name=nerf_lego_highres_lite

Output Visualizations:

Model output of a NeRF (with view dependence) trained on 400x400 resolution:

Model output of a lighter NeRF model trained on 400x400 resolution with fewer parameters (n_hidden_neurons_xyz=64 & n_hidden_neurons_dir=32):

Discussion:

As seen in the GIFs above, on increasing the network resolution the model outputs much more fine-grained details which were absent in the previously trained version. In addition to this, I also experimented with the network capacity of the model to check the impact of the number of training parameters on the results.

We can easily compare the above two outputs and see that the NeRF model with the higher number of parameters synthesizes a much clearer & detailed output as compared to the one with fewer parameters.