| GT | Pred | Point Cloud |
|---|---|---|
| chairs | chairs | |
| chairs | chairs | |
| chairs | lamps | |
| vases | lamps | |
| lamps | vases | |
average classification accuracy of the best model is 97.37
Failure cases are listed in the bottom three point clouds.
The third point cloud (chair) is very flat so the model mistakens it as lamps.
The fourth point cloud (vase) is in a cylinder shape thus is mistakened as lamps.
The fifth one (lamp) has a sphere bottom shape thus is mistakened as vase
These three failure cases happens because the examples are non-typical of their classes, thus misled the model towards wrong predictions.
| GT | Pred | Accuracy |
|---|---|---|
 |
 |
93.46 |
 |
 |
98.60 |
 |
 |
88.59 |
 |
 |
48.44 |
 |
 |
47.27 |
test accuracy of the best segmentaton model is 90.26
The first 3 objects have fairly good segmentation accuracy since each part of the object is clear to classify (slim chair leg, armrest)
The last two sofa objects have poor segmentation performance, since most of the parts in a sofa is flat and the three sides of it are similar and hard to classify.
| num sampled points | classification accuracy |
|---|---|
| 10000 | 97.37 |
| 1000 | 96.33 |
| 100 | 90.77 |
| rotation angle along x-axis | classification accuracy |
|---|---|
| 0 | 97.37 |
| 10 | 96.12 |
| 45 | 58.03 |
| 90 | 24.13 |
| 180 | 69.25 |
we conduct two experiments on the classification task to evaluate robustness
number of sampled points
by decreasing the number of sampled points from 10000 to 100 in test stage, the classification accuracy gradually drops. This is reasonable since more sampled point gives more information
rotation angle along the x-axis
result shows larger rotations angle decrease the performance more (0->10->45->90). This shows the model is less robust on objects with random orientations.
however, when the roration angle is 180, the classification is better than rotation angle of 45 and 90. This is because mirroring the orientation of object (forward to backward) does not affect the model much