The test accuracy of the best model is 0.9748.
The failure cases of the following visualizations show that objects having an elongated shape are misclassified to lamp. Furthermore, I believe that the failure cases in the lamp category support this claim because lamps without long shapes are misclassified to as vase. Therefore, PointNet may not make use of a global feature and predict a category only based on the local feature and shape.
Successful Cases:
Failure Cases:
Both are predicted as a lamp
Successful Cases:
Failure Cases:
Both are predicted as a lamp
Successful Cases:
Failure Cases:
Both are predicted as a vase
The test accuracy of the best model is 0.8991.
According to the visualizations of the failure cases, PointNet struggles with segmenting chair legs and head rest in the case that these areas do not exist or their boarders are ambiguous. However, I think that their segmentation results are still generally consistent (no catastrophic error) and accurate.
test accuracy: 0.9575
test accuracy: 0.9761
test accuracy: 0.8966
test accuracy: 0.8218
test accuracy for 594: 0.8218
test accuracy for 351: 0.4688
test accuracy for 26: 0.4429
test accuracy for 426: 0.4429
test accuracy for 163: 0.6307
I tested the robustness with respect to rotation and the number of samples. For the robustness to rotation, a point cloud is rotaed along with x, y, and z axes by a given rotation angle. For the robustness to the number of sampled points, I simply decrease it by changing the argument of num_points.
The accuracy of all the experiments is reported in the following. Similar tendencies concerning the robustness to rotation and the number of samples are observed for both tasks. Specifically, the results show a significant drop of around 10 to 30 degrees for rotation. I believe that this implies that the classifier heavily counts on the rotational information of an object. PointNet is decently robust to the number of samples. Surprisingly, the accuracy did not change much even though I set it to one-tenth of the original (1000 samples). However, when the number is set to 5, 10, or 100, the accuracy largely deteriorates because it becomes challenging to guess the shape.
0 degrees (Original): 0.9748
5 degrees: 0.96222
10 degrees: 0.8919
15 degrees: 0.6139
20 degrees: 0.4029
25 degrees: 0.2980
30 degrees: 0.2508
35 degrees: 0.2298
40 degrees: 0.2225
45 degrees: 0.2077
50 degrees: 0.2151
55 degrees: 0.2288
60 degrees: 0.2360
65 degrees: 0.2235
70 degrees: 0.2277
75 degrees: 0.3305
80 degrees: 0.3861
85 degrees: 0.4753
90 degrees: 0.6264
10000 samples (Original): 0.9748
5000 samples: 0.9727
1000 samples: 0.9716
100 samples: 0.8017
50 samples: 0.5960
10 samples: 0.2980
5 samples: 0.2644
0 degrees (Original): 0.8990
5 degrees: 0.8790
10 degrees: 0.7457
15 degrees: 0.5568
20 degrees: 0.2897
25 degrees: 0.2504
30 degrees: 0.2272
35 degrees: 0.2389
40 degrees: 0.7124
45 degrees: 0.5185
50 degrees: 0.2457
55 degrees: 0.2441
60 degrees: 0.7557
65 degrees: 0.2554
70 degrees: 0.3933
75 degrees: 0.4071
80 degrees: 0.2090
85 degrees: 0.5979
90 degrees: 0.2429
10000 sampels (Original): 0.8991
5000 samples: 0.8988
1000 samples: 0.8947
500 samples: 0.8866
100 samples: 0.8182
50 samples: 0.7462
10 samples: 0.4587
5 samples: 0.3692