Learning for 3D Vision: Assignement 5

Deepti Upmaka

Q1. Classification Model (40 points)

The accuracy of this network across all samples is: 97.9%

In the case of the failed chair example, it has a long back which looks like a lamp with a long pole. Some of the lamps look like vases since they have the same cylindrical or cube like structure as the vases. This is the same reason why some of the lamps also get predicted as vases. Regardless of these failed cases, the model evaulation accuracy is about 98% which means it is able to predict most of the objects correctly.

Below are a couple of examples of successful predictions

Ground Truth: Chair, Predicted: Chair

Ground Truth: Vase, Predicted: Vase

Ground Truth: Lamp, Predicted: Lamp

Below are a couple of examples of failed predictions

Ground Truth: Chair, Predicted: Vase

Ground Truth: Chair, Predicted: Lamp

Ground Truth: Vase, Predicted: Lamp

Ground Truth: Lamp, Predicted: Vase

Q2. Segmentation Model (40 points)

The accuracy of this network across all samples is: 90.5%

As you can see below there are many examples that perform well with this segmentation network. The examples that don't perform as well are difficult shapes or have more segments than the examples that work well. It is also difficult to predict chair feet since there is a lot of variation between what is considered the feet. The definition of the feet is different depending on if the chair has wheels or which part touches the floor. Another segment that is diffcult to predict are the arm rests. Some chairs dont have them, they might be skinny or they might be completely filled. This is the same problem that is presented in the feet. Below are a few examples of segmentation that was successful.

Ground Truth (left) Predicted (right) Accuracy: 99.75%

Ground Truth (left) Predicted (right) Accuracy: 99.74%

Ground Truth (left) Predicted (right) Accuracy: 99.51%

Ground Truth (left) Predicted (right) Accuracy: 98.86%

Ground Truth (left) Predicted (right) Accuracy: 98.54%

Ground Truth (left) Predicted (right) Accuracy: 98.44%

Below are a few examples of segmentation that was unsuccessful.

Ground Truth (left) Predicted (right) Accuracy: 43.38%

Ground Truth (left) Predicted (right) Accuracy: 50.65%

Ground Truth (left) Predicted (right) Accuracy: 55.11%

Ground Truth (left) Predicted (right) Accuracy: 55.38%

Ground Truth (left) Predicted (right) Accuracy: 55.44%

Q3. Robustness Analysis (20 points)

I conducted two robustness tests.

Rotation: The first one is rotation of the test data while evaluating. Within this I did 3 tests. Overall, there is a great decrease in performance with any kind of rotation so that means this network is not able to handle changes in rotation.
- 90 degrees rotation about the z axis: As expected the accuracy was pretty poor performing which makes sense because the 90 degree changes the general location of the points so it looks like a completely different object.
- 180 degrees rotation about the z axis: For the classification model the 180 degree orientation performed better than the 90 degree rotation because while the points are upside down it retains the basic structure and it can translate in one axis. However for the segmentation model, the 180 rotation actually performed worse than the 90 degree rotation. This interesting but a possible explination could be that the segmentation has a harder time predicting as it deviates from the orginal orientation. So we can say that 90 degrees is closer to the original position than 180 degrees.
- 90 degrees rotation about x axis and 90 degrees rotation about y axis: The comparison for this between segmentation and classification follows the same logic as the previous test. The segmentation performed better in this instance than the other two orientations because it was the least deviated from the orginal position. Whereas for the classification this slight skew greatly decreased the accuracy.

Number of Points: The second is changing the number of points that are used when making the class and segmentation predictions. Within this, I did 2 tests, sampling 2000 and 100 points. Overall, there wasn't much of a difference in the performance with the numbers of points that were sampled. It did decrease, but still was ~90% for classification and ~70% accuracy for segmentation.For both the tests above, for each of the classification and segmentation the accuracy stayed the same regardless of how many points were chosen. This demonstrates that the network is robust with having less information than the number of points the network was trained on.