16-889 16-889 Assignment 5

-1. Number of Late Days

0. Setup

Q1. Classification Model

Quantative Result

My test accuracy of the best model is approximately 97.69%.

Visual Result

	pred = chairs	pred = vases	pred = lamps
label = chairs
label = vases
label = lamps

The total number of points per scene is 10000, and the above table shows the success and failure examples, respectively lying on the diagonal and off-diagonal entries.

At row 2 column 4, a folded chair was recognized as a lamp, and the reason could be that most of the chairs in the training set have a resonably large thickness, and also the lamps there typically have curved tip.

At row 3 column 4, the vase looks quite similar to the cap of a lamp exactly below it in the table. Similarly, the lamp at row 4 column 3 has square cross-section contour, like the vase above it in the table.

The last failure example is the lamp viewed as a chair at row 4 column 2, probably because the lamp has a leg at the bottom , similar to the bases of some chairs.

Q2. Segmentation Model

Quantative Result

My test accuracy of the best model is approximately 90.02%.

Visual Result

visualize segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth, report the prediction accuracy for each object, and provide interpretation in a few sentences.

Bad predictions

index	gt	pred
26
235
605

Good predictions

index	gt	pred
471
488
547

Interpretation

Looking at test 26, the chair only has one side having a significant arm, but the predictor is still looking for two symmetric arms. On test 235, the ground truth seems to be wrong, while the prediction looks more reasonable to me. At last, test 605 is a hard one, because there're no obviously separate arms, and the chair legs here are higher than normal.

On the contrary, the good cases have more common chairs (e.g. legs are under the seat plane), so the accuracies are high over there.

Q3. Robustness Analysis

Q3-1 Reduce num_points

Procedure

Run python eval_cls.py --load_checkpoint best_model --num_points X and python eval_seg.py --load_checkpoint best_model --num_points X, where X could be 10000, 5000, 2500 and so on.

Results

For both classification and segmentation tasks, with 320 points, the accuracies are already very close to the ones in full sampling cases. With just 10 points, classification model totally failed, because the accuracy from a random guess should be 1/3 when there're 3 classes and the classes distributed evenly. While for segmentation task, the accuracy then has already reached 65%, likely because the classes are quite unbalanced. The ratios of the classes in all test points are respectively 0.0033, 0.0014, 0.3197, 0.0913, 0.2031, 0.3813. Actually, if num_points is 0, the segementation accuracy is about 40%, close to 0.3813. Another reason could be that the model has already overfitted this dataset.

Q3-2 Rotate point cloud

Procedure

Add a new rotation matrix variable to rotate point cloud, which can be uniquely defined by new python arguments, roll, pitch and yaw.

Results

According to the results on the cls accuracy column, classification model is quite sensitive to rotation along y axis, and performs better at roll and yaw rotation. These results are not too surprising, but my expectation was that yaw should play the least important role here. As for the segmentation task, accuracies also obviously drops, but the reductions are smaller than the ones in the classification task. Lastly, combined with the results from last section, the segmentation model is more robust against ablation tests.