16-889 Homework2
1.1. Fitting a voxel grid (5 points)
Visualize the optimized voxel grid along-side the ground truth
Generated Target
1.2. Fitting a point cloud (10 points)
Visualize the optimized point cloud along-side the ground truth
Generated Target
1.3. Fitting a mesh (5 points)
Visualize the optimized mesh along-side the ground truth
2. Reconstructing 3D from single view
2.1. Image to voxel grid (15 points)
Image Target Mesh Vox generate
2.2. Image to point cloud (15 points)
Image Target Mesh Point generate
2.3. Image to mesh (15 points)
Image Target Mesh Mesh generate
2.4. Quantitative comparisions(10 points)
Quantitatively compare the F1 score of 3D reconstruction for meshes vs pointcloud vs voxelgrids. Provide an intutive explaination justifying the comparision.
| Voxel | Mesh | Pointcloud |
|---|---|---|
| 76.7 | 89.67 | 90.32 |
Generally, the voxel F1 score is lower than the Mesh and PointCloud, and this could be explained by the loss calculation of the Voxel. The cross entropy can not really reflect the difference between the generated voxel and ground truth. This was also reflected in the training cycle when the loss after epochs become nearly constant. Another factor is possibly the resolution, 32x32x32 could be too big for network to make accurate prediction on small detail. As a contrast, the Point Cloud has the highest F1 score, which fits the expectation that chamfer loss can accurate describe the differences between the pointcloud.
2.5. Analyse effects of hyperparms variations (10 points)
Analyse the results, by varying an hyperparameter of your choice
One hyperparameter tested is the "w__smooth" for the mesh training. If the w__smooth is too small compare to the w__chamfer, the rendered mesh will be very "sharp" and visually more different than ground truth even with high F1 scores. A comparison of result of F1 score and visualization with w_smooth of 0.8 copare to deault 0.1 is listed below.
Ground Truth w_smooth=0.1 (F1=87.6) w_smooth = 0.8(F1=88.2)
2.6. Interpret your model (15 points)
Simply seeing final predictions and numerical evaluations is not always insightful. Can you create some visualizations that help highlight what your learned model does?
The issue with F1 score is even with high F1 score, the predicted result might still look dramatically different compare to the ground truth. To better visually and mathematically show the difference between the two. One method is highlight the "outlier" within a range compare to ground truth. The visualization is presented in point cloud and it accepts meshes and pointcloud as inputs, and if the voxels are sampled, it will also accpet voxel for comparison.This can be further implemented in the network as one of the loss function. The visualized comparison is down below and the red is the ground truth with green as outliers.
Visualization differences based on pointcloud Ground Truth
Lateday: one day late
