Name: Ayush Pandey andrew id: ayushp

Q1. Classification Model (40 points)

To run training and evaluation run

python main.py --question_number q1

Classification Test Accuracy: 97.27%

Sample correct and incorrect predictions:

Chair Correct	Vase Correct	Lamp Correct

Chair predicted as Vase	Vase predicted as Lamp	Lamp Predicted as Vase

Interpretation of the incorrect cases:

While training, the model learns a common spatial arrangement of points across classes i.e. Chair is composed of multiple cuboids which are usually elongated along y direction, vase is a hollow structure and lamp is a structure which connects two concentrations of mass through a thin deformable cylinder. When these assumptions are broken we see an incorrect prediction.

1. Chair predicted as Vase: Its flat along xz axis and has most of its mass concentrated in the top region which might have confused the network.
2. Vase predicted as a Lamp: You can see that its similar to a structure where two masses are connected by a thin deformable cylinder.
3. Lamp predicted as Vase: We can see that its a hollow structure and if inverted woudl look like a vase.

Q2. Segmentation Model (40 points)

To run training and evaluation run

python main.py --question_number q2

Classification Test Accuracy: 90.35%

Sample correct and incorrect predictions:

Correct Prediction	GT

Incorrect Prediction	GT

Incorrect prediction analysis:

Its very difficult for the network to understand about substructures without texture and edge information which is usually absent in point clouds which makes segmentation difficult for the network. We can see that it has high tendency to classify nearby points as the same and where there isn't explicity discontinuity in point cloud structure it fails to change the labels. It does reasonably well on almost all the points and miclassifies points of few objects badly.

Q3. Robustness Analysis (20 points)

1. I rotated the point clouds along all 3 axes for both the tasks, here are the accuracy results: For evaluation run

   python main.py --question_number q3_1

   Classification:

   Angle of Rotation	x axis	y axis	z axis
   60	32.63%	73.66%	31.27%
   120	55.93%	31.58%	23.69%
   180	74.20%	33.68%	51.52%

   Segmentation:

   Angle of Rotation	x axis	y axis	z axis
   60	41.61%	72%	50%
   120	24%	59.8%	33%
   180	35%	59.09%	36%

   As we can see that the model is not robust to rotation. We haven't implemented the transformation blocks and that might have contributed to slight improvement in robustness, but as such the model doesn't have any inductive biases enforced either through dataset or the architecture or the losses to induce rotation invariance.

2. I sampled 10k, 5k, 1k, 100 and 10 points to see how robust the model is to sub sampling, here are the accuracy results.

   For evaluation run

   python main.py --question_number q3_2

   Classification:

   Number of Points	Accuracy
   10k	97.27%
   5k	97.06%
   1k	96.43%
   100	91.18%
   10	24.55%

   Segmentation:

   Number of Points	Accuracy
   10k	90.34%
   5k	90.35%
   1k	90.02%
   100	80.88%
   10	60.17%

   As we can see that the model is quite robust to reducing the number of points sampled. Even when we remove 90% of the points the model is able to retain its accuracy with a drop of max ~10% across segmentation and classification. This is because while by introducing max pooling in the architecture we force the network to compress information from a lot of points which helps it learn from fewer points and discard information from majority of the points which makes it robust to reduction in number of points sampled.