16-889: Assignment 5

Adnan Ahmad

adnana@andrew.cmu.edu

Late Days used - 2

Task 1 – PointNet Classification

PointNet Classification Test Accuracy = 96.85

Good Examples

Point Cloud	Predicted Class
	Chair
	Vase
	Lamp

Failure Cases - Classification

Point Clouds	GT	Predicted
	Vase	Lamp
	Lamp	Vase
	Vase	Lamp

Comments - The results above indicate that the model achieves excellent classification performance by obtaining 97% accuracy. The few cases where it fails have been visualized above. These cases in general seem very hard to classify, even perhaps for most humans. For instance, the second input looks very much like a vase and not a lamp. It’s the same for the other 2 failed cases. Therefore, it can be concluded that the model performs acceptably and only fails in extremely hard to classify cases.

Task 2 – PointNet Segmentation Model

Segmentation Test accuracy = 89.61

Good Examples

GT	Predicted	Accuracy
		99.63
		82.03
		99.18

Failure Cases – Segmentation

GT	Pred	Accuracy
		53.27
		64.37
		63.68

Comments – The pointnet segmentation model achieves 89.6% accuracy on the test set which indicates good performance. The few failure cases have been visualized above. It seems the model fails in cases where different regions of the chair blend into each other without much semantic differentiation. This is clearly in the last example, where the base of the chair merges into the foot, confusing the model. The other 2 cases suffer from a similar problem. Overall, the model still performs really well on most cases as shown by the correctly visualized samples.

Task 3 - Robustness

	Classification	Segmentation	Comments
Num Points			For both the classification and segmentation models I varied the number of points over a large set starting from (0,10000]. I then plotted the test accuracies. My observation is that the accuracy drops significantly for both models if the num of points is extremely less (<50). After that, the accuracy rises and saturates and more or less remains the same, to the effect that adding more points doesn’t yield a noticeable increase in accuracy.
Rotation Angle			For both the classification and segmentation models I rotated the point cloud about the z-axis between [0,360] degrees. For each of these angles I created a rotation matrix using scipy, and took its dot product with the point cloud. I plotted the results vs test accuacy. My observation is that accuracy drops significantly as the model is rotated from its base configuration. This continues until it starts rising again after it starts turning around into its default config (~250 degrees). These plots suggest that the pointnet model is NOT invariant to rotaions and hence additional modifications are required to account for that.