Q1. Classification Model (40 points)

Test Accuracy: $0.973$

0 → Chair | 1 → Vase | 2 → Lamp

Random test point cloud visualization:

Predicted label: Chair | GT: Chair

Predicted label: Chair | GT: Chair

Predicted label: Vase | GT: Vase

Predicted label: Vase | GT: Vase

Predicted label: Lamp | GT: Lamp

Predicted label: Lamp | GT: Lamp

Failure cases:

Predicted label: Vase | GT: Chair

Predicted label: Vase | GT: Lamp

Predicted label: Chair | GT: Vase

Interpretation:

From the randomly selected samples for each class (that are successfully classified), we can see that these points are of more-or-less usual shape belonging to its class (chairs usually have a back supporter, vase are cylinder or box like objects, and lamps usually have a stem and a larger top area). The failure cases, for all three classes, are clear outliers that does not conform to the usual shape (the chair is more flat shaped, the lamps has four irregular heads, and the vase is indeed close to a chair with two legs). In all, the network is successfully capturing the general shapes of the classes, and making mistakes on hard cases that have ambiguous shapes.

Q2. Segmentation Model (40 points)

Overall Test accuracy: $0.909$

5 Object Segmentation:

Pred segmentation; Accuracy: 0.942

GT segmentation

Pred segmentation; Accuracy: 0.995

GT segmentation

Pred segmentation; Accuracy: 0.994

GT segmentation

Pred segmentation; Accuracy: 0.983

GT segmentation

Pred segmentation; Accuracy: 0.987

GT segmentation

2 Bad Segmentation:

Pred segmentation; Accuracy: 0.987

GT segmentation

Pred segmentation; Accuracy: 0.482

GT segmentation

Interpretation: from the visualizations we can see that the segmentation model can easily segment chairs that are of regular shape (in fact, all the high accuracy ones have four legs and one back post). The lower accuracy ones are more irregular, with parts that are hard to classify. Both bad predictions contain a cushion that are not always accompanying the chairs. The first example’s segmentation would even challenge a person. We can also deduct that the high success rate shapes are more prevalent in the training dataset.

Q3. Robustness Analysis (20 points)

1. You can rotate the input point clouds by certain degrees and report how much the accuracy falls

Classification:

Procedure:

I used scipy's rotation library to generate rotations along x-y-z axis to study which axis’s rotation affects the accuracy the most.

Test Accuracy:

Analysis:

x - 45°

y - 45°

z - 45°

x - 90°

y - 90°

z - 90°

From the table we can see that classification error drops significantly when the shape is rotated severely. The more rotation there is, the less the model can classify the shape. Thus, we can see that the model is not super robust to rotation, and is relying on the point cloud’s principle rotation axis (up direction, which is y-axis) for classification. Notably, the y-axis rotation also has the least effect on the test accuracy, which is reasonable since, as can be seen in visualization, the y-axis is the ‘up-vector’ for these chairs and rotating around it does not change the point relationship drastically.

Segmentation:

Procedure:

I used scipy's rotation library to generate rotations along x-y-z axis to study which axis’s rotation affects the accuracy the most.

Test Accuracy:

Analysis:

Similarly to the result in segmentation, we can see that the model, since we do not explicitly handle rotation invariance, is not super robust to rotation. There is a marked drop in model classification accuracy, similar to classification. This is reasonable since our segmentation model also rely on the position of the points and is not rotation invariant. Similarly, we can see that y-axis has the least effect on the accuracy.

2. You can input a different number of points points per object (modify --num_points when evaluating models in eval_cls.py and eval_seg.py)

Classification

Procedure:

In this study, I pass in different number of points to analyze the test accuracy.

Test Accuracy:

Analysis:

50 points: A chair misclassified as lamp

50 points: A vase misclassified as lamp

50 points: A lamp misclassified as vase

100 points: A chair misclassified as lamp

100 points: A chair misclassified as lamp

100 points: A lamp misclassified as vase

1000 points: A chair misclassified as lamp

1000 points: A lamp misclassified as vase

1000 points: A lamp misclassified as vase

From the accuracy progression, we can see that the accuracy of classification saturates around 5000 points, with the biggest jump around 10→ 100 points. This is expected science the more point there is, the more you can make out the shape. However, as the point increases, there is diminishing return in the number of points the model can analyze, and some points may confound the model. 5000 points actually out-perform the more 10000 points case. We can also see that when there is less points (around 10~1000), the model has less points to work with and are making predictions based on the general shape, as expected.

Segmentation

Procedure:

In this study, I pass in different number of points to analyze the test accuracy.

Test Accuracy:

Analysis:

For segmentation, we can see that the test accuracy has less of a jump as the point increases, which is reasonable since classifying different parts of a chair can also rely on the height from the ground (so segmentation is less robust to rotation in our case since we do not model transformation). The accuracy gradually grows as the more points and more global information can be collected.