Assignmnet 5

Name: Neha Boloor, Andrew ID: nboloor

Number of late days used: 0

Download zip file (~2GB) from https://drive.google.com/file/d/1wXOgwM_rrEYJfelzuuCkRfMmR0J7vLq_/view?usp=sharing. Put the unzipped data folder under root directory. There are two folders (cls and seg) corresponding to two tasks, each of which contains .npy files for training and testing.

Q1. Classification Model (40 points)

• Test accuracy of the best model: 0.9758656873032528

• Visualization of a few random test point clouds and the predicted classes for each along with 1 failure prediction for each class :

  Class	Correctly Predicted Example	Incorrectly Predicted Example	Predicted Class Label
  Chair			Vase
  Vase			Chair
  Lamp			Vase

  Interpretation:
  1. The chair instance is misclassified as a vase here, since it is folded and flat in this case and doesn't really have a flat seat area and hence a hard example when compared to the typical chair structure seen commonly in the training set. It is classified as a vase probably due to the cavity between the two planes and the vertical structure.
  2. The vase instance is misclassified as a chair as it has a general structure similar to a chair with a flat seat like area and an upright plane for back support.
  3. The lamp instance is misclassified as a vase as it is quite different from the other lamp structures and has a plant in a vase like structure. Hence it's classified as a vase which is a hard example even for humans to distinguish.

Q2. Segmentation Model (40 points)

• Test accuracy of the best model: 0.9065933549432739

• Visualization of segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth, report the prediction accuracy for each object, and provide interpretation in a few sentences.

  Ground Truth	Prediction	Accuracy
  		0.9948
  		0.9857
  		0.9856
  		0.4762
  		0.4955
  		0.5128

  Interpretation:
  The segmentation network seems to perform well on examples with few prominent classes, that are in some way well separated. As the object becomes more cluttered, the network does not perform as well, perhaps due to structural confusions around those regions as these are just pointclouds (no textures) and distinguishing between points along the class edges along with the intra class variations, becomes hard to learn.

Q3. Robustness Analysis (20 points)

I have conducted 2 experiments to analyze the robustness of your learned model for both cls and seg task.

1. During evaluation we add some noise to the test point cloud i.e. adding jitter to the position of each point by a Gaussian noise with zero mean and varying standard deviation. (sd values tried include 0.01, 0.02, 0.05, 0.3)
Ex: On adding a Gaussian noise with 0 mean and sd = 0.02 (in red) this is what the test sample (in green) gets modified to:
• 
• 
Run the following for this : python eval_cls.py --task cls --add_noise True --load_checkpoint best_model --exp_name cls_noise

Task Type	SD	Test Accuracy
cls	0.01	0.974816369359916
cls	0.02	0.9664218258132214
cls	0.05	0.919202518363064
cls	0.3	0.310598111227702
seg	0.01	0.9024589951377634
seg	0.02	0.8919053484602917
seg	0.05	0.8352265802269043
seg	0.3	0.5454905996758509

This tells us that small corruptions or extra noise points in the input pointcloud are not likely to change the output of our network for a small sd value.
2. Second experiment included varying the input number of points per object.(Values for num_points experimented with include 10,100,1000,5000,10000) Run the following for this : python eval_seg.py --task seg --add_noise True --load_checkpoint best_model --exp_name seg_noise