Daniel Bronstein

Assignment #5

3D Learning

Zero late days used

Part 1

Test accuracy of best model: 91.9%

Accurate Predictions

Point Cloud	Predicted Class
	Chair
	Vase
	Lamp

Inaccurate Predictions

Point Cloud	Predicted Class
	Lamp
	Lamp
	Vase

The offendinging point clouds that lead to innacuracte results are generally atypical examples of the class. In these examples, the chair has a very odd aspect ration relative to typical chairs, the vase is filled with a flower, and the lamp is highly unique. The misclassification of the vase is perhaps the most understandable, since the structure of the flower resembles the head of a lamp.

Part 2

Test accuracy of best model: 87.1%

5 Most Accurate Test Predictions

Ground Truth Classes	Predicted Classes	Accuracy
		99.57%
		99.49%
		99.36%
		99.33%
		99.07%

5 Least Accurate Test Predictions

Ground Truth Classes	Predicted Classes	Accuracy
		30.46%
		41.66%
		42.93%
		44.09%
		51.48%

The accuracy of the segmentation models follows some clear patterns when looking at the best and worst performing pointclouds. Particularly the network works very well with chairs that only include legs, seat and chair-back that have orthogonal connectivity and clearly defined structures. The network failes to perform with irregular shapes and pointclouds that include the other 3 classes, namely arms, ottoman, and headrest. This is resonable given that the lower relative frequency and higher variability of those features. The network can not seem to distinbuish the back and headrest classes, and has trouble distinguishing the seat and ottoman. The ambiguous connectivity of many armrests also proved difficult.

Part 3

Robustness Under Rotation

The model trained on the unaltered dataset was evaluated on pointclouds rotated about the global X axis to varying degrees. The accuracy of the model on the rotated test pointclouds is recorded here.

Classification	Segmentation

The model performs worse with high rotations than it does under 0 rotation, as expected. There is a dip for both classification and segmentation task where the model performs worse than chance (0.33 and 0.17, respectively). This indicates that rotations of certain magnitudes exploit some learned structural features of the model, causing worse than chance performance.

Robustness Under Scaling

The model trained on the unaltered dataset was evaluated on pointclouds scaled about the global origin to varying degrees. The accuracy of the model on the scaled test pointclouds is recorded here.

Classification	Segmentation

The model performs worse when the test set is scaled above or below identity. The dropoff in performance is approximately symmetric in log(scale) space. There is an interesting artifact in the classification performance: when the test dataset is scaled by a factor of 0.5 to 0.6, the classification model suffers significantly. The causal mechanism behind this dip is left as an excercise for the reader.