Deliverables: On your website:
Q. Report the test accuracy of you best model.
A. 0.9696
Q. Visualize a few random test point clouds and mention the predicted classes for each. Also visualize at least 1 failure prediction for each class (chair, vase and lamp), and provide interpretation in a few sentences.
Good Cases
| S.No | Groundtruth Point cloud | Predicted Class |
|---|---|---|
| 1 | |
Chair |
| 2 | |
Chair |
| 3 | |
Vase |
| 4 | |
Lamp |
| 5 | |
Lamp |
Failure Cases
| S.No | Groundtruth Point cloud | Groundtruth Class | Predicted Class |
|---|---|---|---|
| 1 |  |
Chair | Vase |
| 2 |  |
Chair | Lamp |
| 3 |  |
Vase | Lamp |
| 4 |  |
Vase | Lamp |
| 5 |  |
Lamp | Vase |
| 6 |  |
Lamp | Vase |
Interpretation:
From the above results we can observe that the model does pretty well for structures of chairs/ vase/ lamp which capture a common structure. We can observe some failure cases in the table above. Let us dicuss them here:
a) The image 1 has been classified as a vase and that could be due to the round structure which is typically seen in a vase.
b) The image 2 has been classified as a lamp and that could be due the tall structure of the back seat of the chair which is typically seen across different instances of lamp.
c) The image 3 has been classified as a lamp because the flower or leaves seen in the vase resemble the lamp head of a table lamp.
d) The vase in the image 4 resembles a sleek table lamp where the protruding structure of the lamp is similar to the lamp head.
e) In case of images 5 and 6, both of the instances of lamp consists of the curvy structure of a vase. Especially in image 6 the base of the lamp looks a lot similar to a vase.
Q. Report the test accuracy of you best model.
A. 0.9035
Q. Visualize segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth, report the prediction accuracy for each object, and provide interpretation in a few sentences.
Good Cases
| S.No | Groundtruth Point cloud | Predicted Point Cloud | Accuracy |
|---|---|---|---|
| 1 |  |
 |
0.9636 |
| 2 |  |
 |
0.9898 |
| 3 |  |
 |
0.9791 |
| 4 |  |
 |
0.9799 |
| 5 |  |
 |
0.9949 |
Failure Cases
| S.No | Groundtruth Point cloud | Predicted Point Cloud | Accuracy |
|---|---|---|---|
| 1 |  |
 |
0.6363 |
| 2 |  |
 |
0.542 |
| 3 |  |
 |
0.444 |
| 4 |  |
 |
0.6753 |
| 5 |  |
 |
0.5728 |
Interpretation:
As we can see above the segmentation network works pretty well for simple structures where there is clear distinction between different parts of a chair (hand rest, legs, base, back rest, seat).
The model performs really bad most commonly in the case of sofa like structures where there is no clear distinction between the base, the legs and the hand rest. In image 4 we can see how th model is confused to segment the base and the seat of the chair. Even for the human it is quite confusing to make clear distinction between different parts of the chair.
Conduct 2 experiments to analyze the robustness of your learned model. Some possible suggestions are:
--num_points when evaluating models in eval_cls.py and eval_seg.py)Feel free to try other ways of probing the robustness. Each experiment is worth 10 points.
Deliverables: On your website, for each experiment
Experiment 1: Rotating points
Q. Describe your procedure.
I have performed per point rotation by multiplying the rotation matrix (the matrices used are below). Once I have the transformed points I feed it to the model.
Rotation Matrices used:
Angle: 45 degrees
[[ 0.7071, 0.7071, 0.0000],
[-0.7071, 0.7071, 0.0000],
[ 0.0000, 0.0000, 1.0000]]
Angle: 90 degrees
[[ 0.0, 1.0, 0.0],
[-1.0, 0.0, 0.0],
[ 0.0, 0.0, 1.0]]
Angle: 135 degrees
[[-0.7071, 0.7071, 0.0000],
[-0.7071, -0.7071, 0.0000],
[ 0.0000, 0.0000, 1.0000]]
Angle: 180 degrees
[[-1.0, 0.0, 0.0],
[0.0, -1.0, 0.0],
[ 0.0, 0.0, 1.0]]
Q. For each task, report test accuracy of you best model, in comparison with your results from Q1 & Q2
Classification Task
a) Rotation angle: 45 degrees Command to run
python eval_cls.py --rotation --angle 45
Test Accuracy: 0.3337
| Object 1 | Object 2 | Object 3 | Object 4 |
|---|---|---|---|
 |
 |
 |
 |
b) Rotation angle: 90 degrees Command to run
python eval_cls.py --rotation --angle 90
Test Accuracy: 0.2592
| Object 1 | Object 2 | Object 3 | Object 4 |
|---|---|---|---|
 |
 |
 |
 |
c) Rotation angle: 135 degrees Command to run
python eval_cls.py --rotation --angle 135
Test Accuracy: 0.2371
| Object 1 | Object 2 | Object 3 | Object 4 |
|---|---|---|---|
 |
 |
 |
 |
c) Rotation angle: 180 degrees Command to run
python eval_cls.py --rotation --angle 180
Test Accuracy: 0.4270
| Object 1 | Object 2 | Object 3 | Object 4 |
|---|---|---|---|
 |
 |
 |
 |
Interpretation:
As we can see the accuracy of the classification task drops a lot with rotation.
Segmentation Task
a) Rotation angle: 45 degrees Command to run
python eval_seg.py --rotation --angle 45
Test Accuracy: 0.6119
| GT/Pred | Object 1 | Object 2 | Object 3 | Object 4 |
|---|---|---|---|---|
| Groundtruth |  |
 |
 |
 |
| Prediction |  |
 |
 |
 |
b) Rotation angle: 90 degrees Command to run
python eval_seg.py --rotation --angle 90
Test Accuracy: 0.3338
| GT/Pred | Object 1 | Object 2 | Object 3 | Object 4 |
|---|---|---|---|---|
| Groundtruth |  |
 |
 |
 |
| Prediction |  |
 |
 |
 |
c) Rotation angle: 135 degrees Command to run
python eval_seg.py --rotation --angle 135
Test Accuracy: 0.3283
| GT/Pred | Object 1 | Object 2 | Object 3 | Object 4 |
|---|---|---|---|---|
| Groundtruth |  |
 |
 |
 |
| Prediction |  |
 |
 |
 |
d) Rotation angle: 180 degrees Command to run
python eval_seg.py --rotation --angle 180
Test Accuracy: 0.3786
| GT/Pred | Object 1 | Object 2 | Object 3 | Object 4 |
|---|---|---|---|---|
| Groundtruth |  |
 |
 |
 |
| Prediction |  |
 |
 |
 |
Interpretation:
We can observe that the performance of segmentation network drops with rotation. Also one importtant observation is that we can observe how the model has lernt the position of different parts of the chair. For example, the points which lie at the bottom of the point cloud are marked as the base/legs of the chair. Hence in case of 180 degree rotation we can see that the points corresponding to the back rest are marked as legs of the chair(blue).
Experiment 2: Num points
Q. Describe your procedure.
A. The number of points sampled are changed, hence I sampled lesser number of points to check the performance of the model in different tasks.
Q. For each task, report test accuracy of you best model, in comparison with your results from Q1 & Q2
Classification Task
Command to run
python eval_cls.py --num_points 5
python eval_cls.py --num_points 10
python eval_cls.py --num_points 100
python eval_cls.py --num_points 1000
python eval_cls.py --num_points 10000
| num_points | Groundtruth | Prediction | Average Test Accuracy |
|---|---|---|---|
| 5 |  |
Lamp | 0.2476 |
| 10 |  |
Lamp | 0.2644 |
| 100 |  |
Chair | 0.6999 |
| 1000 |  |
Chair | 0.9622 |
| 10000 |  |
Chair | 0.9696 |
Segmentation Task
Command to run
python eval_seg.py --num_points 5
python eval_seg.py --num_points 10
python eval_seg.py --num_points 100
python eval_seg.py --num_points 1000
python eval_seg.py --num_points 10000
| num_points | Groundtruth | Prediction | Average Test Accuracy |
|---|---|---|---|
| 5 |  |
 |
0.6376 |
| 10 |  |
 |
0.6937 |
| 100 |  |
 |
0.8297 |
| 1000 |  |
 |
0.8973 |
| 10000 |  |
 |
0.9035 |
Interpretation:
For both the tasks, the models perform close to the results in Q1/Q2 with 1000 points only. As mentioned in the paper as well, with less number of points as well we are able to capture some basic structure to perform the classification/segmentation tasks.