16-889 Assignment 5

Byeongjoo Ahn (bahn@andrew.cmu.edu)

Late day:

Q1. Classification Model (40 points)

Test accuracy: 97.69%

Correct predictions:

Incorrect predictions:

Interpretation:

While we achieve high test accuracy of 97.69%, the classification network provides incorrect prediction when the input point cloud has uncommon shapes. For example, when the leg of chair has a shape of X instead of general four legs, the network classifies it as a lamp. The additional feature at the bottom part of the lamp also causes the incorrect prediction of lamp. Similarly, when the bottom part of the lamp is flat, the network classifies it as a vase.

Q2. Segmentation Model (40 points)

Test accuracy: 87.24%

Good predictions:

Bad predictions:

Interpretation:

The visualization of good and bad predictions show the top 4 best and worst results. Our network provides accurate results when the target shape is simple (e.g., total number of segments is small) as shown in the good predictions. On the other hand, it provides bad results when the number of segments is large and the target shape has multiple parts including stools.

Q3. Robustness Analysis (20 points)

Q3.1. Robustness to Rotation

We analyze how our model is robust to rotations. We rotate the point cloud by 45, 90 degrees in each axis and compare the test accuracy to the result without rotation.

Classification (97.69% without rotation)

angle \ axis	X-axis	Y-axis	Z-axis
$45^\circ$	40.1	63.06%	67.68%
$90^\circ$	40.82%	47.64%	24.24%

The failure cases for different rotations are visualized. The worst result is obtained when we rotate the point cloud by 90 degrees with respect to the Z-axis. Adding transformation blocks can help the dependency on rotations.

[ $45^\circ$ in X-axis] GT= Chair | Prediction = Vase

[ $45^\circ$ in Y-axis] GT= Chair | Prediction = Vase

[ $45^\circ$ in Z-axis] GT= Chair | Prediction = Vase

[ $90^\circ$ in X-axis] GT= Chair | Prediction = Vase

[ $90^\circ$ in Y-axis] GT= Vase | Prediction = Lamp

[ $90^\circ$ in Z-axis] GT= Chair | Prediction = Vase

Segmentation (87.24% without rotation)

angle \ axis	X-axis	Y-axis	Z-axis
$45^\circ$	67.95%	75.37%	65.44%
$90^\circ$	28.15%	65.24%	43.43%

The best results for different rotations are visualized. When the number of segments is small, we can achieve good test accuracy even when the input point cloud is rotated.

The worst results for different rotations are visualized. The worst results are not obtained when the object shape is irregular (e.g., including stools), which is different with the case without rotation. The segmentation results show that they are segmented just based on the location of the points (e.g., yellow at top, red at middle), not considering the global structure.

Overall, rotating the input point cloud by 90 degrees in a certain axis provide the worst results, but the classification is worst in the rotation w.r.t. Z-axis, and the segmentation is worst in the rotation w.r.t. X-axis. This discrepancy implies good segmentation requires something more than good global feature used in the segmentation. As the current implementation relies on the single global feature for all points, better representation that involves locality can improve the segmentation results.

Q3.2. Robustness to Number of Points

We analyze how our model is robust to the number of points. We change the number of points used for evaluation and compare the test accuracy to the result with the number of points of 10000.

Classification

Number of points	Classification Accuracy
10	24.87%
50	72.40%
100	90.77%
500	96.22%
1000	96.75%
5000	97.59%
10000	97.69%

The failure cases for different number of points are visualized

[#points = 10] GT = Vase | Prediction = Lamp

[#points = 50] GT = Vase | Prediction = Lamp

[#points = 100] GT = Vase | Prediction = Lamp

[#points = 500] GT = Lamp | Prediction = Vase

Segmentation

Number of points	Segmentation Accuracy
10	52.53%
50	74.12%
100	79.24%
500	85.62%
1000	86.88%
5000	87.22%
10000	87.24%

The best results for each number points are visualized. When the number of segments is small, we can achieve good test accuracy even when the number of points are small.

The worst results for each number of points are visualized. When #points > 100, we observe the worst result is obtained when the object shape is irregular (e.g., including stools). However, when the #points is small, point clouds of simple shape show the worst result because of the lack of global information.

Overall, we observe that the test accuracy increases as we increase the number of points both in classification and segmentation. The test accuracy increases rapidly until the number of points becomes 500.