CMU-16889 Learning for 3D Vision - HW5

1. Classification Model

Test accuracy of the best model: 97.48%
Some correct predictions:

Prediction: Chairs

Prediction: Vases

Prediction: Lamps

Some failure cases.

Below is a lamp that is predicted as a vase. Since many vases in training data are along with flowers inside, the two light bulbs on top of long rods do have a similar pattern as flowers inside a vase if the model only consider the point coordinates.

Below is a chair that is mostly predicted as a lamp and sometimes as a vase. It doesn't seem to have any similar pattern as a lamp or a vase. Perhaps this is not similar to other chairs in the training data, so the model makes a mistake.

Below is a vase that is predicted as a lamp. It is understandable that the model can get confused with some vases and lamps because of the reason discussed in the first failure case. In this one, the curve pattern of the leaf (I guess) is similar to a kind of a light bulb of a lamp.

2. Segmentation Model

Test accuracy of the best model: 90.61%. In each image pair below, Left: Ground truth. Right: Prediction.
Some correct predictions and each's accuracy:

Accuracy: 94.08%

Accuracy: 97.05%

Accuracy: 94.86%

Some failure cases and each's accuracy. provide interpretation in a few sentences.

Accuracy: 77.94%
The model has a bad segmentation at the lower cusion and the model thinks it's chair legs. In most of the training data, the lower part of the whole chair is chair legs. Without the additional information aside from point location, it is expected that under the current model design, the model can't distinguish between chair legs and cusion in this example.

Accuracy: 62.72%
The model thinks the sides of the cusion are chair arms and the bottom are chair legs. This example, as the left example, has a less "regular" shape. Without further infomration, such as some geometry, the model has little information about the definition of chair arms and chair legs. It's decision pretty much relies on the point's relative location with respect to the whole object.

3. Robustness Analysis

Experiment 1

For each object, I rotate the the point cloud by a random angle about the z=axis during evaluation to test the model's robustness against rotation. The model get only 34.84% for the classification task and get 48.22% for the segmentation task. It is expected that the accuracy drops but this drop is pretty significant. Obviously, the model is not rotation invariant and equivariant.

Experiment 2

I sample less number of points during evaluation to test the robustness of the model. I set the number of points to be 5000, 1000, 500, and 100. We can see that a larger number of points gets higher accuracy than a smaller number of points. The reason is perhaps more points can capture more information of the object.
5000 points: 97.38% for the classification task and 90.58% for the segmentation task.
1000 points: 96.96% for the classification task and 90.13% for the segmentation task.
500 points: 95.59% for the classification task and 88.88% for the segmentation task.
100 points: 85.10% for the classification task and 81.36% for the segmentation task.