Some random test point clouds with their label and predcition (0: chairs, 1: vases and 2:lamps)
Label: 0, Prediction: 0
Label: 1, Prediction: 1
Label: 0, Prediction: 0
Label: 1, Prediction: 2
Label: 2, Prediction: 2
Some failure cases with their label and predcition (0: chairs, 1: vases and 2:lamps)
Label: 0, Prediction: 2
Label: 1, Prediction: 2
Label: 2, Prediction: 1
Label: 2, Prediction: 0
Label: 0, Prediction: 1
Q2. Segmentation Model (40 points)
The test accuracy of my best model is 89.26%.
Some random predictions (top row) and the corresponding ground truth (bottom row)
Object 246 prediction with an accuracy of 96%
Object 30 prediction with an accuracy of 97%
Object 436 prediction with an accuracy of 95%
Object 85 prediction with an accuracy of 82%
Object 97 prediction with an accuracy of 61%
Object 246 GT
Object 30 GT
Object 436 GT
Object 85 GT
Object 97 GT
Some bad predictions (top row) and the corresponding ground truth (bottom row)
Object 163 prediction with an accuracy of 28%
Object 235 prediction with an accuracy of 53%
Object 26 prediction with an accuracy of 52%
Object 351 prediction with an accuracy of 54%
Object 96 prediction with an accuracy of 54%
Object 163 GT
Object 235 GT
Object 26 GT
Object 351 GT
Object 96 GT
Q3. Robustness Analysis (20 points)
Classification Task
I first test the robustness by rotating each point cloud by a random degree (below the max_rot angle) for each of the x, y, z axis.
The quantitative results are shown below:
Max Random Rotate Angke (degree)
Accuracy (%)
0 (Q1)
95.70
30
65.06
60
36.52
120
31.37
180
31.69
Some qualitative results and comparsions are shown below (no rotation points in the top row, random rotation under 60 degrees in the bottom row):
Label: 0, Prediction: 0
Label: 1, Prediction: 1
Label: 0, Prediction: 0
Label: 1, Prediction: 2
Label: 2, Prediction: 2
Label: 0, Prediction: 2
Label: 1, Prediction: 2
Label: 0, Prediction: 2
Label: 1, Prediction: 2
Label: 2, Prediction: 2
I then test the robustness by changing the number of points per object. The results are shown below:
Number of Points
Accuracy (%)
10000 (Q1)
95.70
1000
95.49
100
93.39
10
72.29
Some qualitative results and comparsions are shown below (10000 points in the top row, 100 points in the bottom row):
Label: 0, Prediction: 0
Label: 1, Prediction: 1
Label: 0, Prediction: 0
Label: 1, Prediction: 2
Label: 2, Prediction: 2
Label: 0, Prediction: 0
Label: 1, Prediction: 2
Label: 0, Prediction: 0
Label: 1, Prediction: 2
Label: 2, Prediction: 2
Segmentation Task
I first test the robustness by rotating each point cloud by a random degree (below the max_rot angle)
for each of the x, y, z axis. The results are shown below:
Max Random Rotate Angke (degree)
Accuracy (%)
0 (Q2)
89.26
30
74.08
60
56.87
120
38.96
180
35.17
Some qualitative results and comparsions are shown below (no rotation points in the top row, random rotation under 60
degrees in the second row, ground truth in the bottom row):
Object 246 prediction with an accuracy of 96%
Object 30 prediction with an accuracy of 97%
Object 436 prediction with an accuracy of 95%
Object 85 prediction with an accuracy of 82%
Object 97 prediction with an accuracy of 61%
Object 246 prediction with an accuracy of 71%
Object 30 prediction with an accuracy of 58%
Object 436 prediction with an accuracy of 46%
Object 85 prediction with an accuracy of 13%
Object 97 prediction with an accuracy of 56%
Object 246 GT
Object 30 GT
Object 436 GT
Object 85 GT
Object 97 GT
I then test the robustness by changing the number of points per object. The results are shown below:
Number of Points
Accuracy (%)
10000 (Q2)
89.26
1000
88.16
100
80.10
10
62.43
Some qualitative results and comparsions are shown below (10000 points in the top row, 100 points in the second row, ground truth in the bottom row):
Object 246 prediction with an accuracy of 96%
Object 30 prediction with an accuracy of 97%
Object 436 prediction with an accuracy of 95%
Object 85 prediction with an accuracy of 82%
Object 97 prediction with an accuracy of 61%
Object 246 prediction with an accuracy of 91%
Object 30 prediction with an accuracy of 96%
Object 436 prediction with an accuracy of 88%
Object 85 prediction with an accuracy of 58%
Object 97 prediction with an accuracy of 44%
Object 246 GT
Object 30 GT
Object 436 GT
Object 85 GT
Object 97 GT
As we shall observe in these results, the performance drops as the input is rotated for both classification and segmentation.
This is because the model does not have rotation invarience. Thus, as the test data is rotated, it deviates from the training data distribution and makes the
model hard to predict correctly. Meanwhile, the performance also drops as the number of points is decreased. This is because fewer number of points mean there
are too few details to describe the point cloud for the model to make a good prediction.
Q4. Bonus Question - Locality (20 points)
I implemented the DGCNN model. Since DGCNN is taking much GPU memeory, I compute and compare the perfromance of
DGCNN and PointNet under the condition where num_points = 1000 (different than num_points = 10000 in Q1&Q2).
The accuracies and comparisions with PointNet are shown.
Task
Method
Accuracy (%)
Classification
PointNet
95.28
Classification
DGCNN
96.22
Segmentation
PointNet
88.52
Segmentation
DGCNN
89.87
Some qualitative results and comparsions of the classification task are shown below (PointNet in the top row, DGCNN in
the bottom row):
Label: 0, Prediction: 0
Label: 1, Prediction: 1
Label: 0, Prediction: 0
Label: 1, Prediction: 2
Label: 2, Prediction: 2
Label: 0, Prediction: 0
Label: 1, Prediction: 1
Label: 0, Prediction: 0
Label: 1, Prediction: 1
Label: 2, Prediction: 2
Some qualitative results and comparsions of the segmentation task are shown below (PointNet in the top row, DGCNN in the second row, ground truth in the bottom row):
Object 246 prediction with an accuracy of 96%
Object 30 prediction with an accuracy of 97%
Object 436 prediction with an accuracy of 95%
Object 85 prediction with an accuracy of 75%
Object 97 prediction with an accuracy of 61%
Object 246 prediction with an accuracy of 97%
Object 30 prediction with an accuracy of 97%
Object 436 prediction with an accuracy of 96%
Object 85 prediction with an accuracy of 87%
Object 97 prediction with an accuracy of 70%
Object 246 GT
Object 30 GT
Object 436 GT
Object 85 GT
Object 97 GT
As we can observe, the performance of DGCNN is better than the PointNet because it reasons about the local feature of the point cloud.
