Point Clouds: Assigment 5 (16-889)

Name: Shefali Srivastava

Andrew ID: shefalis

Late Days:

Q1. Classification Model (40 points)

Test Accuracy

Accuracy: 0.974816369359916

Visualisations

S.No.	Input Point Cloud	Grouth Truth Class	Predicted Class
1		Chair	Chair
2		Chair	Chair
3		Chair	Lamp
4		Vase	Vase
5		Vase	Vase
6		Vase	Lamp
7		Lamp	Lamp
8		Lamp	Lamp
9		Lamp	Vase

Interpretation

I’ve visualised 3 predictions for every class in order of chair, vase and lamp with 2 success and 1 failure case for each class:

• Chair

Row 1, 2: Input Chair / Prediction Chair

Row 3: Input Chair / Prediction Lamp

I think this could have happened because the chair in row 3 is elongated in the horizontal direction whereas most other chair are taller.

• Vase

Row 4, 5: Input Vase / Prediction Vase

Row 6: Input Vase / Prediction Lamp

It is not difficult for the model to confuse a lamp and a vase. A vase is broader on the bottom while a lamp is taller and elongated. In Row 6, the vase actually is of the same cylindrical dimension from top to bottom as opposed to most vases that have varying radiuses along its height. I believe this can be the reason why the model was unable to classify it correctly.

• Lamp

Row 7, 8: Input Lamp / Prediction Lamp

Row 9: Input Lamp / Prediction Vase

Similar to the reason for 2, I believe that the model associates a wider bottom narrowing and then widening into a top with a vase like structure. This lamp is structurally very similar to a vase and that is why the model misclassified it in my opinion.

Q2. Segmentation Model (40 points)

Test Accuracy

Accuracy: 0.8876335494327391

Visualisations

S.No.	Ground Truth	Prediction	Prediction Accuracy (%)
1			52.32
2			73.53
3			74.41
4			79.4
5			89.42
6			91.9
7			94.48
8			94.66

Interpretation

I have visualised chairs in sorted order of their accuracy. The worst 2 performing chairs are on the first two rows and the best performing chair is on the last row. I believe that most chairs have a back-rest, two handles and legs. In the first two rows, the chairs seems to firstly be a couches/sofa chairs and secondly also contain a red segmented cushion. This is perhaps what has caused confused with the model as it is unable to segment that area of a couch cushion separately. The best performing chairs are the ones with well defined segments most representative of the maximum type of chairs in the dataset.

Q3. Robustness Analysis (20 points)

1. Rotating Input Point Clouds:

CLASSIFICATION

Quantitative Comparisons

Visualisations

S.No.	Input Point Cloud	Rotation (30)	Rotation (60)	Rotation (90)
1
2
3

Interpretation

Rotation along Z Axis:

rotation_matrix = torch.Tensor([
        [torch.cos(theta_rads).item(), -torch.sin(theta_rads).item(), 0],
        [torch.sin(theta_rads).item(), torch.cos(theta_rads).item(), 0],
        [0, 0, 1]
    ])

I tried experimenting with rotation of the input point clouds. I rotated them by 30, 60 and 90 degrees along the Z axis and as expected, the accuracy falls in that order itself. This is expected since a particular item stops looking like itself as it keeps on rotating. While the model still latches on to the correct prediction for lower degrees of rotation, once the item is rotated by 90 degrees, the model does start to make evident mistakes. I have visualised one row for correct prediction and one row for prediction that was correct but gradually becomes incorrect (for all three classes).

1. Chair

   ** S.No: 1, Row 1: The input chair gets incorrectly predicted as the rotation degrees increase. This is probably because the chair is very symmetrically distributed and once it rotates by 90 degrees, it stops looking like a chair and becomes wider spread instead of being elongated, like a lamp or a vase and this is reflected in the predictions as well.

   ** S.No: 1, Row 2: A very distinctive chair with longer legs and well-defined hand-rest continues to be classified correctly due the these features.

2. Vase

   ** S.No: 2, Row 1: The input vase clearly has flowers. While one flower could have been mistaken by the network as being a bulb-holder in the lamp, multiple flowers are quite suggestive of a vase and the model correctly predicts all throughout the rotations.

   ** S.No: 2, Row 2: I am not surprised that for this one the vase starts to be predicted as a lamp. There are no distinctive features and a wider bottom could equally be associated with a lamp as well.

3. Lamp

   ** S.No: 3, Row 1: This one is a tricky for the model. A wider bottom, narrower lead to the top and then a slightly wider top is most suggestive of a vase.

   ** S.No: 3, Row 2: The model has latched on to the definition where a bulb-holder is tall and thin over a wider base, as suspected. Is it able to classify such a lamp correctly even on rotation.

SEGMENTATION

Quantitative Comparisons

Visualisations

S.No.	Input Point Cloud	Rotation (0)	Rotation (30)	Rotation (60)	Rotation (90)
1
2
3
4
5
6

Interpretation

Similar to classification, the accuracy falls as the rotation degrees increase. Most seat (red) segments are in the middle, the legs are at the bottom (dark blue) segment and the back-rest are (light blue) segments. Unless the back-rest are very well defined (as in the last two rows in the visualisation), the model tries to associate the middle section with mostly seat (red) segments (first two rows). Beyond that, the model seems to perform quite good in my opinion since till the second last rotation row (other than 90 degrees), it is able to form most logical segments.

2. Altering Number of Points in a Point Cloud:

CLASSIFICATION

Quantitative Comparisons

Visualisations

S.No.	Input Point Cloud	Num. Point Clouds (10000)	Num. Point Clouds (5000)	Num. Point Clouds (2500)	Num. Point Clouds (1250)	Num. Point Clouds 625)	Num. Point Clouds (312)	Num. Point Clouds (156)	Num. Point Clouds (78)	Num. Point Clouds (39)	Num. Point Clouds (19)	Num. Point Clouds (9)	Num. Point Clouds (4)	Num. Point Clouds (2)	Num. Point Clouds (1)
1
3
5

Interpretation

I have computed and visualised point clouds at varying number of points in a point cloud as shown in the table. Till 78 points, the accuracy does not really fall a lot, does not even go below 90. This is indicative of the fact that the only a subset of the provided points constitute what the model believes as helpful for classification. Going down from 78 points the accuracy starts to see a visible hit all the way down to 1 point at which the model predicts randomly (with 1/3 = 33%) accuracy for each.

SEGMENTATION

Quantitative Comparisons

Visualisations

S.No.	Input Point Cloud	Num. Point Clouds (10000)	Num. Point Clouds (5000)	Num. Point Clouds (2500)	Num. Point Clouds (1250)	Num. Point Clouds 625)	Num. Point Clouds (312)	Num. Point Clouds (156)	Num. Point Clouds (78)	Num. Point Clouds (39)	Num. Point Clouds (19)	Num. Point Clouds (9)	Num. Point Clouds (4)	Num. Point Clouds (2)	Num. Point Clouds (1)
1
2
3
4
5
6

Interpretation

Similar to classification, the accuracy falls at 78 points in a point cloud at which it becomes difficult for the model to predict.