16889 Learning For 3D HW5 Webpage

Akshay Hinduja, ahinduja@andrew.cmu.edu, late days used: 3

Q1 Classification Model

The best model reported an accuracy of 0.975 (epoch ~200)

A few examples of the successful classfications from each class are as follows for chair, vase and lamp

And a few examples of incorrectly classfied as

Ground-Truth:Prediction
Chair:Lamp

Vase:Lamp

Lamp:Vase

While the chairs have the least misclassified objects, some are ambiguous like the folding chair here. ,br />

Some of the objects are very different from majority of the trained models. The misclassified Vase has a very different structure and its top part could be misclassified as a lamp and even a the back of chair.

The lamp has a vase like structure so it's understandable why such a classificition can be made.

Q2 Segmentation Model

The accuracy of the best segmentation model is 0.902 (epoch ~200)

3 succesful segmentations with accuracy are reported as follows (GT up, Pred down):

Number 18, accuracy 0.98
Number 23, accuracy 0.953
Number 49, accuracy 0.8958

Two bad classifications are as follows (GT up, Pred down):

Number 61, accuracy 0.604

Number 500, accuracy 0.778

The more complex chairs like no 61 and 500 are difficult due to not as well defined features like the arm rests beldning with the seat, or the legs and seat blending. Other than that the model is able to give good segmentation results.

Q3 Robustness Analysis

Classification model

Analysis of num_points on classification:

10000 points, accuracy 0.975
1000 points, accuracy 0.967
100 points, accuracy 0.941
50 points, accuracy 0.840
30 points, accuracy 0.660 , misclassified as lamp
30 points, accuracy 0.285, misclassified as lamp

Analysis of rotation on classification:

0 deg, accuracy 0.975
15 deg, accuracy 0.954
30 deg, accuracy 0.863
45 deg, accuracy 0.618, misclassified as lamp
60 deg, accuracy 0.385, misclassified as lamp
90 deg, accuracy 0.242, misclassified as vase

The model is not robust to rotation. It is seen that as long as recognisable features like the back of the chair and seat are in a familiar orientation it is good, but once that changes the predictions get inaccurate.

Segmentation model

Analysis of num_points on segmentation:

10000 points, accuracy 0.902
1000 points, accuracy 0.900
100 points, accuracy 0.810
50 points, accuracy 0.746
30 points, accuracy 0.674
30 points, accuracy 0.520

Analysis of rotation on segmentation:

0 deg, accuracy 0.902
15 deg, accuracy 0.840
30 deg, accuracy 0.739
45 deg, accuracy 0.643
60 deg, accuracy 0.469
90 deg, accuracy 0.278

The segmentation model is more resilient towards the number of points compared to rotation. We see that rotations can quickly disrupt prediction accuracy where as for number of points the change is gradual.