Assignment 5

Q1. Classification Model

For classification task, I trained the model for 150 epochs in total, and the best model is stored at 99 epoch. The best test accuracy is 0.980.

I randomly picked some point clouds as shown below. Their predictions are: chair, vase, lamp (they are all correct predictions).

There are also some incorrect predictions. My best model works perfectly on the chair class. There is no incorrect prediction of chair, so I choose the incorrect samples on the previous saved model (epoch = 50). Also, I evaluate the model with 2000 points.

The predictions are: lamp, lamp, vase. The ground truths are: chair, vase, lamp. From the pointclouds, we can see that the incorrect predictions have some similar shape to other classes. For example, the vase one is actually similar to the lamp shape, which makes the predictor confused. Also, similarly, the lamp one looks like a vase. The chair one actually is a little bit confusing since from the point cloud, it is not like the normal chair we might see.

 

Q2. Segmentation Model

For segmentation task, I trained the model for 150 epochs in total, and the best model is stored at 123 epoch. The best test accuracy is 0.898.

I randomly picked some point clouds as shown below.

The prediction accuracy: 0.940.

The prediction accuracy: 0.986.

The prediction accuracy: 0.953.

The prediction accuracy: 0.733.

The prediction accuracy: 0.768.

The last two predictions are bad predictions. From the raw accuracy number, we can see their accuracies are far below the average. The first one has a low accuracy since it makes wrong prediction on the chair legs. The structure of the chair makes the predictor confused about the legs. Without knowing the ground truth, the prediction also makes sense to me. The second one has low accuracy because of the unclear boundary between each parts. It might be hard for the predictor to know the real boundary.

 

Q3. Robustness Analysis

3.1 Rotation

To analyze the robustness of the models I trained, I applied a 90 degrees rotation on all the test points for both tasks (classification and segmentation). Specifically, I multiplied all the test points with the rotation matrix along z axis.

Classification

The accuracy of my best model is only 0.245. This makes sense to me since the model I implemented is not transformation invariant (does not include transformation block). The big drop on the accuracy also proves that. With big rotation, the predictor cannot recognize the shape. Below are some prediction results and their rotated pointcloud input.

 

Segmentation

The accuracy of my best model is only 0.401. Below are some prediction results and their rotated pointcloud input. From the predictions, I can see that the model is still trying to segment the object along the same direction as before. It did not adjust the segmentation based on the object's orientation. Therefore, the accuracy is far below the one in Q2.

The accuracy: 0.326

The accuracy : 0.449

 

3.2 Number of Sampling Points

To analyze the robustness of the models I trained, I also tried with different number of points for evaluation. For both tasks, I experimented with 10000 points, 5000 points, 1000 points, 500 points, and report their accuracy.

Classification

The accuracy is shown below:

• 10000 points (best model in Q1): 0.980
• 5000 points : 0.981
• 1000 points : 0.977
• 500 points : 0.979

With less and less sampling points, the shape of the objects become less clear to the model. As a result, the prediction accuracy drops as I decreased the number of points.

 

Segmentation

The accuracy is shown below:

• 10000 points (best model in Q2): 0.898
• 5000 points : 0.898
• 1000 points : 0.892
• 500 points : 0.879

Similary to the classification task, with less and less sampling points, the structure of the objects become less clear to the model. Also, since we have less points in total, one mistake in prediction would makes the final accuracy to be even smaller. As a result, the prediction accuracy drops as I decreased the number of points.