The model is trained for a total of 30 epochs. All the other parameters are default.
| Test Accuracy |
|---|
| 94.85% |
Correct Predictions:
| Test Point Cloud | Prediction | Ground Truth |
|---|---|---|
 |
Chair | Chair |
 |
Chair | Chair |
 |
Vase | Vase |
 |
Vase | Vase |
 |
Lamp | Lamp |
 |
Lamp | Lamp |
Wrong Predictions:
| Test Point Cloud | Prediction | Ground Truth |
|---|---|---|
 |
Lamp | Chair |
 |
Lamp | Vase |
 |
Lamp | Vase |
 |
Lamp | Vase |
 |
Vase | Lamp |
Overall, the model performs really well with an accuracy of about 95%. From the analysis of wrong predictions, it becomes clear that the model is good at classifying chairs, but confuses vase and lamp. This could be attributed to how vase and lamp look very similar, and have a huge diversity in shapes.
The model is trained for a total of 30 epochs. All the other parameters are default.
| Test Accuracy |
|---|
| 89.45% |
Good Predictions:
| Prediction | Ground Truth | Accuracy |
|---|---|---|
 |
 |
98% |
 |
 |
95% |
 |
 |
90% |
Bad Predictions:
| Prediction | Ground Truth | Accuracy |
|---|---|---|
 |
 |
51% |
 |
 |
54% |
 |
 |
63% |
 |
 |
70% |
 |
 |
73% |
From the failure cases, it seems that the segmentation model struggles with sofa-like chair, and chairs with a distinctively unique shape. Additional complexities like a pillow leads to undesirable results. The model really works well with well-defined chairs with thin surfaces and parts.
Please utilize robustness_cls.py and robustness_seg.py for running these experiments.
For this experiment, a random subset of point are selected from the input point cloud. The number of points argument --num_points determines the number of points selected.
Classification Model
| Number of Points | Test Accuracy |
|---|---|
| 10000 | 94.85% |
| 1000 | 94.64 % |
| 500 | 94.12% |
| 100 | 93.28% |
| 50 | 90.87% |
| 10 | 67.99% |
Segmentation Model
| Number of Points | Test Accuracy |
|---|---|
| 10000 | 89.45% |
| 1000 | 88.82% |
| 500 | 87.54% |
| 100 | 79.2% |
| 50 | 73.4% |
| 10 | 59.83% |
As expected, the classification model is really robust to the number of points. Even on decreasing the number of points to 50, the model perfoms fairly well (90.87%). Downsampling the point cloud by even 100 times leads to almost the same accuracy (93.28%) as compared to using all points (94.85%).
Similar behaviour can be observed with the segmentation model. However, downsampling 100x hurts the model performance, but utilizing 500 points leads to almost the same accuracy (87.54% vs 89.45%).
For this experiment, a random noise is added to the input point cloud. Specifically, we sample random 3D noise vectors from a Gaussian distribution with zero mean and variance given by --noise_var argument. Then, these noise vectors are added to the input point cloud.
Classification Model
| Noise Variance | Test Accuracy |
|---|---|
| 0.001 | 94.85% |
| 0.01 | 94.75% |
| 0.1 | 90.13% |
| 0.5 | 24.55% |
Segmentation Model
| Noise Variance | Test Accuracy |
|---|---|
| 0.001 | 89.44% |
| 0.01 | 89.10% |
| 0.1 | 75.85% |
| 0.5 | 48.10% |
Both the classification and segmentation models are fairly robut to small changes in the input point cloud. But, on increasing the variance, the performance sharply drops as expected.