Part 1

The best test accuracy for my model (in 80 epochs) is: 0.9695697796432319

I visualize a few random test point clouds and mention both the ground truth (GT) and predicted classes (Pred).

The below row are correct predictions.

GT: Chair | Pred: Chair	GT: Vase | Pred: Vase	GT: Lamp | Pred: Lamp

The below rows are wrong predictions.

The chairs may be wrongly classified as Vase if they are too rectangular. Chairs may also be wrongly classified as lamp in case the legs are too thin.

The vase are often mistaken as lamp since they look too similar.

The lamp may be misclassified as vase as they look similar.

Wrong Prediction: GT: Chair | Pred: Vase	Wrong Prediction: GT: Chair | Pred: Lamp
Wrong Prediction: GT: Vase | Pred: Chair	Wrong Prediction: GT: Vase | Pred: Lamp
Wrong Prediction: GT: Lamp | Pred: Vase

Part 2

Report the test accuracy of you best model: 0.8678756888168557

visualize segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth, report the prediction accuracy for each object, and provide interpretation in a few sentences:

I included the two bad samples at the first two rows. Note that these are atypical chairs (e.g., sofa), so the learned model cannot generalize to these unseen types of chair.

Bad sample 1 GT	Bad sample 1 Pred | Prediction Acc: 0.3791
Bad sample 2 GT	Bad sample 2 Pred | Prediction Acc: 0.4263
Good sample 1 GT	Good sample 1 Pred | Prediction Acc: 0.9764
Good sample 2 GT	Good sample 2 Pred | Prediction Acc: 0.9827
Good sample 3 GT	Good sample 3 Pred | Prediction Acc: 0.9658

Part 3

(A) Rotate the input point clouds by certain degrees and report how much the accuracy falls

Procedure: I used https://www.andre-gaschler.com/rotationconverter/ to generate rotation matrix that rotate the point cloud around the z-axis by various radians (0.1 and 0.5 radian).

For Classification:

Original Acc without rotation: 0.9695697796432319

Acc after rotation of 0.1 radian (the accuracy barely falls): 0.9622245540398741

Acc after rotation of 0.5 radian (the accuracy falls alot): 0.6988457502623295

For Segmentation:

Original Acc without rotation: 0.8678756888168557

Acc after rotation of 0.1 radian (the accuracy barely falls): 0.863323662884927

Acc after rotation of 0.5 radian (the accuracy falls alot): 0.7328476499189627

(B) Different number of points points per object

Procedure: I input a different number of points points per object (modify --num_points when evaluating models in eval_cls.py and eval_seg.py)

For Classification:

Original Acc: 0.9695697796432319

Acc reducing to 100 points (the accuracy falls by a bit): 0.9265477439664218

Acc reducing to 50 points (the accuracy falls even more): 0.7775445960125918

For Segmentation:

Original Acc: 0.8678756888168557

Acc reducing to 100 points (the accuracy falls by a bit): 0.8115883306320908

Acc reducing to 50 points (the accuracy falls even more): 0.7462884927066451