Submitted by Ambareesh Revanur (arevanur)
Number of late days used is 0.
For the following answers, the left figure is the ground truth mesh and the right figure is the 3D prediction after overfitting.
Image 1
Image 2
Image 3 (limitation of my model: performs poorly on thin structures)
Image 1
Image 2
Image 3 (the input image has complicated structure with text and shifted chair)
Image 1
Image 2
Image 3 (the input image has complicated structure with text and shifted chair)
Using the eval_model.py, I obtained the following results.
Voxels: 70.1
Point cloud: 89.4
Mesh: 80.9
Comparison: The voxel space is limited to 32x32x32 which is insufficient to model fine structures. Although point cloud results are good, it does not generalize well on rare chairs (e.g. with text) in the test set.
I believe that Mesh's results can be improved by training the model with a deeper network. In my study, I limited the decoder network to simple FC layers. Further, there is scope to change hyperparameters to obtain better results.
| Hyperparams | w_chamfer |
w_smooth |
F1@0.05 |
|---|---|---|---|
| Config 1 | 0.01 | 10 | 80.9 |
| Config 2 | 0.01 | 1 | 84.9 |
| Config 3 | 0.01 | 100 | 75.4 |
From left to right --> Image, Ground Truth 3D mesh, Config 1 predicted 3D mesh, Config 2 predicted 3D mesh, Config 3 predicted 3D mesh
Config 1
Config 2
Config 3
Conclusion When we increase w_smooth keeping everything else fixed (architecture, other hyperparameters) the model predicts smoother models at the cost of a drop in F1score. Similarly, if we decrease w_smooth the model's F1 score improves but the resulting predictions look less smooth.
The voxels that are rendering by using marching cubes is limited to a particular levelset. In order to fully understand the behavior of model, we need to check the soft boundaries.
This is achieved by plotting transparent voxels in 3D grid based on the confidence of the model. If the model is very confident about the occupancy, then this visualization scheme will reduce the transparency (more opaque). Similarly, the model is have a very low confidence, then this visualization scheme will increase the trasparency.
Image 1: We can see that model is making a low confidence prediction around the wheels. This could help in determining the right thresholds for rendering voxels.
I have integrated the code into eval_model.py and render_utils.py. This piece of code will run while evaluating voxels in eval_model.py.
Image 2
Please run python fit_data.py --type impl. Left figure is ground truth in voxel representation and right figure is the predicted implicit function for the chair.
