Assignment-5: Pointcloud Classification and Segmentation

• AndrewID: nlakshmi
• Num of late days: 3

---

Q1. Classification Model (40 points)

• report the test accuracy of you best model.

  python eval_cls.py
  test accuracy: 0.9769150052465897
  

• visualize a few random test point clouds and mention the predicted classes for each. Also visualize at least 1 failure prediction for each class (chair, vase and lamp), and provide interpretation in a few sentences.

  Some random sample pointclouds of correct predictions

  Pred Chair	Pred Vase	Pred Lamp

  Some random sample pointclouds of wrong predictions

  Pred Chair	Pred Vase	Pred Lamp
  GT=Lamp	GT=Chair	GT=Vase

  Observations

  • From the above results, we can see that the model is learning the generic shapes of each of the classes. ie, a chair as for legs and a flat surface, the vase has a cylindrical bottom, and lamps are tall and thin frames.

  • The wrong predictions clearly show this interpretation. The vase is predicted as a lamp here since it seems to be a tall objet and has no "leafy" pattern. The wrong predictions for chair and vase in the first 2 colums is quite confusing to interpret myself. All chairs generally have some depth, but the example in this wrong classification is quite narrow.

Q2. Segmentation Model (40 points)

• report the test accuracy of you best model

  python eval_seg.py
  test accuracy: 0.9035912479740681
  

• visualize segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth, report the prediction accuracy for each object, and provide interpretation in a few sentences.

  GT	Pred	Acc
  		0.9974
  		0.9966
  		0.996
  		0.4816
  		0.4971
  		0.5148

Observations

• The above table shows 3 results each for the best and worst predictions.
• The best predictions are for simple chair objects that have a seating area, back, and legs clearly identified.
• The bad results show that the model is getting confused for the cushion and side rests, especially for these kind of chairs, which is still valid as they look similar.
• the legs are always identified at the bottom correctly. This indicates the model is probably learing the spatial orientaion of these classes, ie, legs are at the bottm, seat/cusion is in the middle/above..etc.

Q3. Robustness Analysis (20 points)

Classification Task

• Rotation

  Procedure - Rotated each object by a certain degrees along the z-axis and evaluated the accuracies.

  python eval_cls.py --rotate x
  

  Angle	Acc
  0	0.9769150052465897
  45	0.670514165792235
  90	0.3284365162644281
  135	0.17103882476390347
  180	0.5739769150052466
  225	0.21720881427072403
  270	0.23504721930745015
  315	0.6484784889821616

• Reduced points

  Procedure - Each object was sampled by a certain number of points and evaluated the accuracies as shown below.

  python eval_cls.py --num_points x
  

  Num Points	Acc
  10	0.5383001049317944
  20	0.7534102833158447
  30	0.8205666316894019
  40	0.8803777544596013
  50	0.9045120671563484
  100	0.9422875131164743
  500	0.9643231899265478
  1000	0.9695697796432319
  5000	0.9769150052465897
  10000	0.9769150052465897

Segmentaiton Task

• Reduced Points

  Procedure - Each object was sampled by a certain number of points and evaluated the accuracies as shown below.

  python eval_seg.py --num_points x
  

  Num Points	Acc
  10	0.5567260940032415
  20	0.6346839546191247
  30	0.6821717990275526
  40	0.7166531604538088
  50	0.7406482982171799
  100	0.8074716369529984
  500	0.8873938411669368
  1000	0.8990194489465154
  5000	0.9035948136142625
  10000	0.9035912479740681

• Rotation

  Procedure - Performed this example for a single object rotated by a centain degrees around z-axis and evaluated the accuracies as shown below.

  python eval_seg_robust.py --i 0 --rotate x
  

  Angle	GT	Pred	Acc
  0			0.935
  45			0.7869
  90			0.4372
  135			0.2029
  180			0.2189
  225			0.2291
  270			0.4713
  315			0.6318

Observations:

• For classification, we can see a big jump in accuracy from 10 to 20 points, but a much slower raise in accuracy for higher number of points. 500 points seems to be sufficient to get 96% accuracy. 10000 point increases the accuracy by only 1%.

• However, for segmentation, there is a linear corelation between the number of points and accuracy. More the number of points, better is the accuracy. Even for this tasks, 5000 points seems to be result in same accuracy as with 10000 points, and there is only a 2% increase in accuracy between 500 points and 10000 points. This indicates that for this model, 500 points seeems to be sufficiet to learn the object features and perfrom reasonably well on both tasks.

• While the model seems to be quite robust to reduced number of points, the results indicate that the model does not perform as well with change in orientation in both tasks.

• The last experiment with single object shows that the model is learning only learning the position of each of the classes (legs, cushion, back, etc) w.r.t to the original orientation of the input data. When rotation is introduced, the model fails as it does not seem to have learnt the position and orientation of each of the class w.r.t to the position of the other classes, instead of just its position as an individual part in 3D space.

Q4. Bonus Question - Locality (20 points)

Incorporate certain kind of locality as covered in the lecture (e.g. implement PointNet++ or DGCNN, etc).

Deliverables: On your website,

• specify the model you have implemented
• for each task, report the test accuracy of your best model, in comparison with your results from Q1 & Q2
• visualize results in comparison to ones obtained in the earlier parts

---

Last update: April 18, 2022