Overview

Zero late days.

Q1. Classification Model (40 points)

Run python train.py --task cls to train the model, and python eval_cls.py for evaluation.

Test Accuracy: 0.952

Correct Cases:

Point Cloud	Predicted Label	Ground Truth Label
	Lamp	Lamp
	Lamp	Lamp
	Vase	Vase
	Chair	Chair
	Chair	Chair

Failure Cases:

Point Cloud	Predicted Label	Ground Truth Label
	Chair	Lamp
	Chair	Lamp
	Chair	Lamp
	Vase	Lamp
	Lamp	Vase

Interpretation:

The first three examples were classified as chair even though they were lamps. Even though they to us look like lamps, the model is unable to classify them as such. The third example is a bit harder, and the network gets confused in predicting chair since it is such a wide object. The last two examples, the model seems to have some confusion between vase and lamp. Both objects do somewhat look like cuboids which could be either a vase or a lamp. The network isn't understanding the plant

Q2. Segmentation Model (40 points)

Run python train.py --task seg to train the model. Running python eval_seg.py will save two gif's, one for ground truth and the other for model prediction. Check out the arguments and feel free to modify them as you want. In particular, you may want to specify --i and --load_checkpoint arguments in eval_seg.py to use your desired model checkpoint on a particular object.

Test Accuracy: 0.890

Correct Cases:

Ground Truth Point Cloud	Predicted Segmented Point Cloud	Accuracy
		0.998
		0.995
		0.994
		0.994
		0.991

Failure Cases:

Ground Truth Point Cloud	Predicted Segmented Point Cloud	Accuracy
		0.52
		0.56

Interpretation:

The segmentation output struggles in identifying common areas of parts of the object. As shown above the model failed to predict the class of the bottom of the couch for both examples. Furthermore, the armrest / side of the couch extends to the back, and can be somewhat arbitrary with just point input and no information about texture.

Q3. Robustness Analysis (20 points)

Task 1: Rotation

The first task I applied was rotation along the Z axis to the input point cloud. I attempted different rotations from 0 to 180 degrees and visualized the results below.

Example Outputs:

Rotation (in degrees)	Ground Truth Point Cloud	Predicted Segmented Point Cloud
0
10
30
60
90
120
180

Rotation (in degrees)	Classification Accuracy	Segmentation Accuracy
0	0.952	0.890
10	0.933	0.861
30	0.571	0.707
60	0.247	0.550
90	0.209	0.428
120	0.231	0.316
180	0.498	0.376

Interpretation:

Our model is clearly not invariant to rotation. As shown above, rotation significantly harms both classification and segmentation accuracy. Specifically, in the examples above we can see that our segmentation model seems to segment points purely based on the height along the chair.

Task 2: Scaling

The second task I applied was scaling the points. I took the distance of each point from the mean point of the object, and increased it by some scale factor. This resulted in the object getting bigger or smaller. I show the results below similar to above.

Example Outputs:

Scale Factor	Ground Truth Point Cloud	Predicted Segmented Point Cloud
0.25
0.5
1.0
1.5
2.0

Scale Factor	Classification Accuracy	Segmentation Accuracy
0.25	0.219	0.367
0.5	0.600	0.594
1.0	0.952	0.890
1.5	0.767	0.770
2.0	0.658	0.677

Interpretation:

Our model is clearly not invariant to scaling. As shown above, scaling significantly harms both classification and segmentation accuracy. Specifically, in the examples above we can see that our segmentation model seems to segment points purely based on the height along the chair. It is important to note that this could be avoided by normalizing the point cloud or applying additional normalization layers inside the model.