Homework4 learning for 3D

Contact Information:

• Name: Shibo Zhao
• Email: shiboz@andrew.cmu.edu
• Late days: using 2 late days

#ipython nbconvert --to HTML your_notebook.ipynb

1. Sphere Tracing

Sphere tracing is a tool to render the implict surface based on ray cast method. Rays are generated from the image pixels and are martched along the specific direction from near to far until it reaches the surface.

The main advantage of sphere-tracing compared to ray-marching is its ability to move along the ray using some sort of adaptive step-size that is automatically computed by the algorithm, removing the need of having to manually choose a step size.

Visualization

To run the code:

python main --config-name=torus

Spiral Rendering of Part 1

Implementation Detail

To render Tours, there are three parameters to contorl the performance of the render: far distance, near distance and max iteration. The tracing process will be repeated before until reaching the maximum iteration. Usually, we assume the origins of rays should be always generated "outside of surface" instead of inside. Therefore, the SDF of rays should be always positive. We set the converged_threshold=1e-5, which means if the SDF value is smaller than the converged_threshold, the ray will be marked as valid.

2. Optimizing a Neural SDF (30pts)

To run the code:

python main --config-name=points

Visualization

Spiral Rendering of Part 2

Implementation Detail

We train a Neural SDF based on the given point cloud. A important note is SDF of point lies on the surface should be 0. We directly use the network from Assignment3. The big difference is that we only use positional harmonic embedding and no view dependance. Our MLP architecture include the harmonic embedding layer, 6 fully connected layers with Relu, 1 fully connected layers and 1 fully conntected layers with a 1 dimension output.

The Eikonal loss is implemented as following.

  gradient_error = (torch.linalg.norm(gradients, ord=2, dim=-1) - 1.0) ** 2
  gradient_error=gradient_error.mean()

We also evalute the infuluence of different loss for final rendering performance. For the whole optimization, we have three losses

loss = loss(distances, sdf_gt) # L1 sdf loss
loss += torch.exp(-1e2 * torch.abs(eikonal_distances)).mean() * cfg.training.inter_weight # distance loss
loss += eikonal_loss(eikonal_gradients) * cfg.training.eikonal_weight # gradient of eikonal_loss

Using all the loss	Using L1 sdf loss and distance loss	Using L1 sdf loss and gradient loss

In this experiment, we found that the all the losses are necessary and directly incluence the performance of rendering. We can not negelect any loss otherwise we cannot obtain the correct bunny shape.

3. VolSDF (30 pts)

Implementation Detail

To run the code:

python main --config-name=volsdf

1. Q: How does high beta bias your learned SDF? What about low beta?

Low $\beta$ will make the density sharper and less smooth. The high $\beta$ tends to flatten the density and make the density more smooth.

2. Q: Would an SDF be easier to train with volume rendering and low beta or high beta? Why?

High $\beta$ might be more eaiser for trainning since the high beta reprenets that the density is more flatten and it will avoid the loss to zero. Otherwise, if the $\beta$ is low, the exponential will be close to zero. We cannot learn any useful information through this loss.

3. Would you be more likely to learn an accurate surface with high beta or low beta? Why?

It would be more likely to create an accuracte surface with a lower $\beta$ since the the low $\beta$ will generates a greate density difference between outside and inside of surfaces. Therefore, it will generate the larger penalties if the loss is large and force the sharp and accurate surface

Results

Geometry Model	Visual Rendering

4. Neural Surface Extras (CHOOSE ONE! More than one is extra credit)

4.2 Fewer Training Views (10 pts)

Results

Loss curve (Training views=100)	Visual Rendering (VolSDF)	Visual Rendering (Nerf)

Loss curve (Training views=20)	Visual Rendering (VolSDF)	Visual Rendering (Nerf)

Loss curve (Training views=5)	Visual Rendering (VolSDF)	Visual Rendering (Nerf)

Implementation Detail

I reduced the number of training views to 20 and 5. As expected, the VolSDF result get better result with 100 views compared with 20 and 5 training views. Compared the trajectory of trainng loss, the VolSDF with 100 views is more easier to coverge to small number with less epoch.

Also, we found that the performance of Nerf with 100 views is better than VolSDF. In spite of this, we found that the performance of Nerf with 20, 5 views are worse than VolSDF. I think the major reason is that the Nerf using the HarmonicEmbedding(3, cfg.n_harmonic_functions_dir) while the Volsdf disable that function. Therefore, when the training view are sufficient, the NeRF can obtain the better result. However, the Nerf is more sensitve than Volsdf with fewer training views.

4.3 Alternate SDF to Density Conversions (10 pts)

Implementation Detail

I tested the other ways to convert the SDF to density.

Method1 $density=abs(sdf)$

Method2 if sdf<=0, $density=\alpha(0.5+0.5\exp(-\frac{(sdf)}{\beta}))$ .

However, if sdf>0 $density=\alpha(0.5-0.5\exp(-\frac{(sdf)}{\beta})) $

Method1 is worse than VolSDF and Method 2. Meanwhile, the performance of method 2 looks simiar to VolSdF method since I just did some small modification based on the VolSDF.

Results

Geometry (Method1)	Visual Rendering (Method1)

Geometry (Method2)	Visual Rendering (Method2)

Geometry Model (Volsdf)	Visual Rendering (Volsdf)