Corinne Alini HW 5

I have used 0 late days on this assignment

Q1. Classification Model

Test accuracy: 97.69%

This model was reasonably accurate. There are a couple failures cases shown below. The first chair that was predicted as a lamp could be poorly predicted because of its height. It is a rather tall chair that resembles the thin shape of a lamp. The chair that was predicted as a vase could be as it is rather ornate and decorative in shape. The vase that was incorrectly predicted as a lamp This is the confusion matrix for the system. As you can see lamps and vases were often confused by the system. This could be because they are both ornate and decorative.

Chair	Vase	Lamp
614	1	2
0	91	11
0	8	226

Successful cases:

Chair Successes

Vase Successes

Lamp Successes

Chair Failures

This chair was predicted as a lamp.
Question 1

This chair was predicted as a vase
Question 1

Vase Failures

This vase was predicted as a lamp.
Question 1

Lamp failures

This lamp was predicted as a vase
Question 1

Q2. Segmentation Model

Test accuracy: 90.38%

Shown below is some of the best and worst performing segmentations. Segmentation worked best when the chair looked similar to a char without arms The square chair had problems as it had problems segmenting the seat from the arms. The arms were hard to segment from the seat it seems.

Successful chair: 98.99%

Successful Vase: 97.55%

Successful Lamp: 98.99%

Failure:45.2%

Q3. Robustness

Number of points

The first thing I tested was decreasing the number of points. I tested 3 different decreases of points, 200, 2000 and 5000.

Classification

5000 points: 97.58 2000 points: 97.37 200 points: 95.59

5000 points confusion table

Chair	Vase	Lamp
614	1	2
0	91	11
0	9	225

2000 points confusion table

Chair	Vase	Lamp
614	1	2
0	90	12
0	10	224

200 points confusion table

Chair	Vase	Lamp
614	1	2
2	82	18
0	19	215

In this case, reducing the number of points reduced the classification accuracy for all cases. There was a direct relationship between number of points and the accuracy of classification. This makes sense as we have less information to classify.

Successful vase example 10000 points

Failure vase example 5000 points

Failure vase example 2000 points

failure vase example 200 points

Segmentation

5000 points: 90.41 2000 points: 90.32 200 points: 84.23

In this case, reducing the number of points reduced the segmentation accuracy for all cases. There was a direct relationship between number of points and the accuracy of segmentation. However, given that the accuracy didnt drop by much, the network is rather robust to reducing the number of points.

Successful chair example 5000 points 99.04%

Successful chair example 2000 points 98.55%

Successful chair example 2000 points 93.5%

Rotation

I rotated the images 90 degrees about the z-axis. I did so by multiplying the test data by

0	-1	0
1	0	0
0	0	1

then fed them through the network. This method worked poorly for classification. Our accuracy was worse than previous. It is not robust to rotations. I tested 30,45 and 90 degree rotations to determine how much worse it got. This makes sense as the network might have been able to figure out segments that are close to the original orientation but not as it got farther and farther from the training data.

Classification

30 degree rotation: 77.85%
45 degree rotation: 68.73%
90 degree rotation: 52.15%

Confusion matrix for 30 degree rotation

555	5	7
9	84	9
81	50	103

This was misclassified as a vase
Question 1

___________________________________
Confusion matrix for 45 degree rotation

492	108	17
23	74	5
81	64	89

This was misclassified as a chair
Question 1

___________________________________
Confusion matrix for 90 degree rotation

324	236	57
8	58	36
43	76	115

This was misclassified as a vase
Question 1

Segmentation

30 degree rotation: 83.31%
45 degree rotation: 69.99%
90 degree rotation: 40.76%

30 degree rotation

Gt, predicted
Question 1

45 degree rotation

Gt, predicted
Question 1

90 degree rotation

Gt, predicted
Question 1