16-889: Learning for 3D Vision
Assignment - 5
Aditya Ghuge aghuge
Late days
used: 0
Q1. Classification Model (40 points)
Test accuracy of best model
: 0.9727
Correct Predictions
|
Class |
Image 1 |
Image 2 |
Image 3 |
|
Chair |
|
|
|
|
Vase |
|
|
|
|
Lamp |
|
|
|
Failure Cases
|
Failure Class |
Ground Truth Class |
Prediction |
Image |
|
Chair |
Chair |
None |
|
|
Vase |
Vase |
Chair |
|
|
Vase |
Lamp |
|
|
|
Lamp |
Lamp |
Vase |
|
Explanation:
For vase being misclassified as chair it can be seen in vases to have a top portion
of plant which is not present in the current pointcloud
and also the pointcloud has
a flat surface on top which resembles sitting surface of a chair.
For
Vase being misclassified as lamp. Lamp and vase have a common pattern of a lean
vertical branch followed by flower(in vase) or lamp(in
lamp) at the top. Although both the patterns are a bit distinct but have this
similarity which can cause the model to misclassify one class to other. Most of
the misclassification of my model were these misclassifications.
For
Lamp being misclassified as Vase: same as above.
Q2. Segmentation Model (40 points)
Test accuracy of best model
: 0.9020
Segmentation Visualization:
|
Ground Truth |
Prediction |
Accuracy |
|
|
|
0.3673 |
|
|
|
0.456 |
|
|
|
0.50 |
|
|
|
0.9468 |
|
|
|
0.9663 |
|
|
|
0.9679 |
|
|
|
0.9907 |
Interpretation: The model is trained well to
segment points of chair shapes which fall in the category of “standard” chair
shape per say ie. Which has a vertical backrest, sitting
cushion(horizontal flat surface) and legs(vertical
single or 4legs or with wheels). The model performs poor on chair shape which
defy this standard shape such as displayed above like in image 3 there is not
concrete concept of vertical support or legs for chair and evidently model
performs poorly on those areas of pointclouds.
Similar trend can be observed with image 2 where there are 2 sub classes for
vertical support. Also if there is
presence of extra part of chair associated it with(leg
rest in image 1) the model fails to categorize it as different class and causes
poor segmentation.
Q3. Robustness Analysis (20 points)
Experiment 1: Changing Number of Points
In this simple approach I will just
change(reduce) the number of input points provided to the model for testing the
performance of model with low number of points. As we have results for num_points = 10000, I am performing experiment with num_points = 10, 50, 100, 500, 1000, 5000
CLASSIFICATION RESULT
|
Number of Points |
Test Accuracy |
|
10000 (Original) |
0.9727 |
|
10 |
0.6516 |
|
50 |
0.9087 |
|
100 |
0.9307 |
|
500 |
0.9560 |
|
1000 |
0.9664 |
|
5000 |
0.9706 |
As evident from the results above the model
performs sufficiently well even with 1/100th (100 points acc: 90+) of
the input test data and is above 90% but falls to 65% with just 10points which
is understandable as 10 points is insufficient to describe a shape of object.
SEGMENTATION RESULT
|
Number of Points |
Test Accuracy |
|
10000 (Original) |
0.9020 |
|
10 |
0.6587 |
|
50 |
0.7544 |
|
100 |
0.8014 |
|
500 |
0.8743 |
|
1000 |
0.8905 |
|
5000 |
0.9018 |
As evident from the results above the model
performs sufficiently well even with 1/100th (100 points acc: 89+) of
the input test data and is above 89% but falls to 75% with just 50points which
is understandable as 50 points is insufficient to describe a shape and
structure of object.
Experiment 2: Rotating the test data
In this experiment I have kept the number of
points constant, but I have rotated all the input points along x axis in 30,
60, 90 and follow the same principle along y axis and z axis. Following are the
results.
CLASSIFICATION RESULT
|
Axis Rotation |
Degree of Rotation |
Test Accuracy |
|
None |
0 |
0.9727 |
|
X-axis |
30 |
0.7891 |
|
60 |
0.2046 |
|
|
90 |
0.36 |
|
|
Y-axis |
30 |
0.9066 |
|
60 |
0.8174 |
|
|
90 |
0.8121 |
|
|
Z-axis |
30 |
0.7586 |
|
60 |
0.4743 |
|
|
90 |
0.2865 |
As evident from the results model is not
invariant to rotation of pointclouds. Though it performs
reasonably well on rotation of objects along Y-axis as compared to X or Z axes.
Also as the angle of rotation increases the performance
of the model decreases.
SEGMENTATION RESULT
|
Axis Rotation |
Degree of Rotation |
Test Accuracy |
|
None |
0 |
0.9727 |
|
X-axis |
30 |
0.7588 |
|
60 |
0.5928 |
|
|
90 |
0.3183 |
|
|
Y-axis |
30 |
0.7919 |
|
60 |
0.6860 |
|
|
90 |
0.5662 |
|
|
Z-axis |
30 |
0.6101 |
|
60 |
0.5042 |
|
|
90 |
0.3835 |
As evident from the results model is not
invariant to rotation of pointclouds. Though it performs
reasonably well on rotation of objects along Y-axis as compared to X or Z axes.
Also as the angle of rotation increases the performance
of the model decreases.