Assignment 3

Paritosh Mittal (paritosm)

Late Days used : 2 days

1.3 Ray Sampling (Visualization)

Grid	Ray

1.4 Point Sampling (Visualization)

Sampling Visualization

1.5 Volume Rendering

Rendered cube gif	Depth

2.3 Basic Implicit Volume (Visualization)

Optimized Implicit Volume

3. Optimizing NeRF

Rendered Lego NeRF with no view dependence

4.1 View Dependence

I add view dependence to previous NeRF implementation. To enable view dependence, please set the view_dep option to true in corresponding config. Adding view dependece enables the model to learn view-dependent effects such as glossiness/specularities. I first visualize the NeRF representations trained with and without view dependence.

Rendered Lego NeRF without view dependence	Rendered Lego NeRF with view dependence

It is interesting that both renderings from implicit representations look very similar. I would generally expect the view-dependent represention to capture details better. However, I suspect the low resolution nature makes identifying such artifacts difficult.

There is a general trade-off between increased view dependence and generalization quality. Specifically, if too much emphasis is given to view dependence then it may considerably impact the learnt implicit representation, such that its rendering look natural when close to observed views (overfitting) but fails to capture the scene geometry. This impacts renderings from novel views.

I know visualize two high resolution implicit representations with and without view dependence.

High Res Rendered Lego NeRF without view dependence	High Res Rendered Lego NeRF with view dependence

This visualization aligns well with our expectation that view dependence improves (to some extent) the learnt representation. More specifically, the lighting (or shadow) changes abruptly from NeRF w/o view dependence (left) with this transition is smooth and natural in NeRF w view dependence (right). Also the lego board on the bottom looks better with view dependence. More details on modified hyper-parameters is in section 4.3

4.2 Hierarchical Sampling (EC)

I did not implement this part of the assignment

4.3 High Resolution Imagery (EC)

Network Capacity:

I made the following changes to HighRes config: Changed n_layers_xyz from 6 to 8 and added an additional skip connection at layer 6. Additing view dependence improved results and hence it is set as default behavior. I visually compare the performance with using n_hidden_neuron size as 128 vs 256. For this experiment 128 points are sampled per ray.

High Res Rendered Lego NeRF with 128 hidden dim	High Res Rendered Lego NeRF with 256 hidden dim

It is evident from this visualization that improving model capacity greatly improves the learnt NeRF representation. The gif in right has captured finer details and is more crisp as compared to the one in left.

Point Samples per ray:

Next, I compare the performance with sampling 64, 128 and 196 points per ray. For this experiment hidden dim is 256.

High Res Rendered Lego NeRF with 64 point samples per ray	High Res Rendered Lego NeRF with 128 point samples per ray	High Res Rendered Lego NeRF with 196 point samples per ray

Left to right, the visualization indicates that increasing point samples per ray greatly improves the optimized NeRF representation. However, this is also coupled with increased memory requirements while 'learning'. We observe that NeRF learnt with 196 samples per ray has crisper structures (example features from bottom lego surface is captured better)