16-889 Assignment 5

Name: Sri Nitchith Akula
Andrew ID: srinitca

Q1. Classification Model (40 points)

Run Command

python train.py --task cls 
python eval_cls.py 

Test Accuracy

test accuracy: 0.9737670514165793

Correctly Predicted Ouputs

Idx	Point Cloud	GT	Pred
0		chair	chair
622		vase	vase
720		lamp	lamp

Wrongly Predicted Outputs

Idx	Point Cloud	GT	Pred
445		chair	vase
716		vase	lamp
803		Lamp	vase

Interpretation for the wrong predictions

• We can see from the first and third image that the network could be predicting taller objects to be vase.
• In the second image, its difficult even for a human to tell whether it is vase or lamp. We can see that it takes shape of some flower and so it could be wrongly classified as vase.

Q2. Segmentation Model (40 points)

Run Command

python train.py --task seg 
python eval_seg.py 

Test Accuracy

test accuracy: 0.9031212317666126

Idx	GT	Pred	Acc
26			0.4753
255			0.4821
351			0.4883
397			0.9945
562			0.994
471			0.9951
297			0.9982

Interpretation

• From the segmented colors, we can notice the following correspondances : red - seat/cushion, blue - legs, light blue - back rest, yellow - arm rest
• The first 3 images are the poorly performing charis with low segmentation accuracy.
• In the first image, we can see that the segmentation output has two arm rests. Since most of the chairs are symmetrical, it could be trying to predict yellow (arm-rest) on the other side as well
• In the second image, it is not clear for the network to differentiate between the seat and the legs as they are continuous and there is no shape change
• In the third image, it is hard to classify whether the back cushion to be considered as cushion or back rest. Also, since most of the chairs have thin legs below the cushion, it could be segmenting the below points to be blue.
• In the last four images, we can see the network is segmenting all the points well.

Q3. Robustness Analysis (20 points)

Experiement 1 : Evaluation for different number of points

Classification

Run Command

python  eval_cls.py --num_points xxxx

Segmentation

Run Command

python  eval_seg.py --num_points xxxx

Interpretation

• In case of classification, decreasing the number of points to 200 is still able to give reasonable accuracy (> 0.9). This means that even if we have sparse point cloud, the network could classify the object reasonably well
• In case of segmentation, the accuracy remains almost same till 1000 points. We can infer the same observation that you do not need a dense point cloud to segment reasonably well.
• We can say that both the networks are robus to number of points till 1000 points. If we go below that, it could affect the accuracy

Experiement 2 : Evaluation for different rotations along z-axis (rotation in x-y plane)

Classification

Run Command

python  eval_cls.py --rot xxxx

Rotation Angle (in degrees)	Sample Point cloud	Accuracy
0		0.9737
30		0.5341
60		0.2612
90		0.2444
120		0.2235
150		0.2539
180		0.4837
210		0.2665
240		0.2444
270		0.2413
300		0.2623
330		0.5414
360		0.9737

Segmentation

Run Command

python  eval_seg.py --rot xxxx

Rotation Angle (in degrees)	Sample GT	Sample Pred	Accuracy
0			0.903
30			0.745
60			0.600
90			0.438
120			0.319
150			0.303
180			0.334
210			0.305
240			0.364
270			0.466
300			0.557
330			0.691
360			0.903

Interpretation

• Rotation has a big impact on both classification and sementation accuracy
• Its easier to visualize the impact of rotation on segmentation. We can see that the network is trying to classify the bottom points as blue and top points as light blue irrespecive of the orientation/rotation of the chair. This is because during training, the network saw examples of chair which are only up-right