Assignment 5

Name: Manikandtan Chiral Kunjunni Kartha

Andrew ID: mchiralk

Late days used: 0

Q1. Classification Model (40 points)

Run python train.py --task cls --load_checkpoint best_model to train the model, and python eval_cls.py --load_checkpoint best_model for evaluation.

All results in this section are from using the best model with 10000 points.

Test Accuracy

test accuracy: 0.9821

visualizations - correct

ID	GT class	Detected Class
600	0 (chair)	0
700	1 (vase)	1
800	2 (lamp)	2

visualizations - failures

For the given model with 10000 points, there were no errors for class 0 (chair). Samples of error in other 2 classes are visualized here.

ID	GT class	Detected Class
618	1 (vase)	2
667	1 (vase)	0
827	2 (lamp)	1

Interpretation

The classifier seems to work well in most cases with a 98% accuracy. The failures (when the confidence of the incorrect class in higher), seems to happen only for objects for ambiguous shapes. In the first sample, the vase looks similar to some hanging lamps in the training set. Simliarly, the second sample is visually similar to stool-like chairs and the third row lamp is very similar to vase shapes. The classifier seems to do a good job except when in cases where the point cloud shape isn't good enough to humanly distinguish the classes.

Q2. Segmentation Model (40 points)

Run python train.py --task seg --load_checkpoint best_model to train the model, and python eval_seg.py --load_checkpoint best_model for evaluation

All results in this section are from using the best model with 10000 points.

Test Accuracy

test accuracy: 0.9012

visualizations - correct

ID	GT segmentation	Predicted Segmentation	Match Ratio
48			0.9912
99			0.9611
215			0.9913
282			0.9591

visualizations - failures

ID	GT segmentation	Predicted Segmentation	Match Ratio
26			0.5454
255			0.5368
595			0.5262
605			0.5402

Interpretation

The segmentation algorithm seems to work well in most cases with a 90% accuracy. The failures (the points with the lowest per point classification match), seems to happen only for objects with ambiguous boundaries. In row 1, the head and seating sections are quite contiguous and the classes overextend without clear demarcations. Similarly, in row 2, the bottom section has no visual demarcation and the algorithm has a hard time distinguishing it from the seating area. For rows 3 and 4, similar effects are seen where the connecting area between the sides, seating and bottom sections are fluid and provides no visual cues for a clear demarcation, and hence have low matching scores.

Q3. Robustness Analysis (20 points)

Experiment 1: Rotation

I rotated the point cloud in steps of 30 degrees on the XY plane and ran classification and segmentation evaluation. The following are the accuracies:

Classification

python eval_cls.py --load_checkpoint best_model --num_points --exp_name < exp_name> Number of points: 1000

Rotation Angle	sample GT visualization -idx 1	Accuracy
0		0.972
30		0.813
60		0.278
90		0.334

Interpretation

We see lower accuracy with increasing rotation angles. This shows the variation in the training data isn't good enough to handle such test time distribution shifts.

Segmentation

python eval_seg.py --load_checkpoint best_model --num_points --exp_name < exp_name> Number of points: 1000

Rotation Angle	Accuracy
0	0.892
30	0.782
60	0.559
90	0.227

sample idx	Rotation Angle	Match Ratio
1	0	0.972
1	30	0.813
1	60	0.278
1	90	0.334

Interpretation

We see lower accuracy with increasing rotation angles. Areas with a large number of points are classified better than thin structures without much defining features. The whole segmentation accuracy breaks down for rotations more than 60 degrees. This maybe improved by adding augmentations in the training set.

Experiment 2: Number of Points

I used the number of input points as the hyperparameter and ran classification and segmentation evaluation. The following are the accuracies:

Classification

python eval_cls.py --load_checkpoint best_model --rot_angle --exp_name < exp_name>

Number of points	sample GT visualization - idx 1	Accuracy
50		0.775
200		0.949
1000		0.972
5000		0.976
10000		0.982

Interpretation

We see lower accuracy for low number of points, which is expected. with 50 points, visual distinction of object class is difficult. As there are only 3 classes in the training set, the classifer still works considerably well for 200 points and above.

Segmentation

python eval_seg.py --load_checkpoint best_model --rot_angle --exp_name < exp_name>

Number of Points	Accuracy
50	0.662
200	0.824
1000	0.892
5000	0.900
10000	0.901

sample idx	Number of Points	Match Ratio
1	50	0.84
1	200	0.975
1	1000	0.983
1	5000	0.983
1	10000	0.986

Interpretation

We see lower accuracy for very low number of points, which is expected. Overall the confidence is higher, with more available input points.

Q4. Bonus Question - Locality (20 points)

N/A