16889-Assignment 5

Jinkun Cao (jinkunc@andrew.cmu.edu)

Q1

Through my implementations, the best accuracy from the first 30 epochs is 0.9685 on the test set.

1. There are some visualizations output on which my model gets the right prediction.

2. Also, there are a failure case for each category:

The first's true category is vase but is predicted to be lamp. The second's true category is chair but is predicted as lamp. The true category of the last one is lamp but is predicted to be vase.

For the first one, it is predicted as lamy by wrong may result from its slim contour which is very common for lamp instances. The second is predicted to be lamp instead of the true label chair, a potential reason is that its being folded, not similar to most chair instances. The third one is very hard, it is just a regular cylinder, making even myself hard to determine it is a vase or a lamp.

Q2

Among the first 30 epochs, the best test accuracy is 0.8984.

I include the visualizations together with the correspondng GTs and the prediction accuracy below.

#100 GT prediction | acc=0.816 #200 GT prediction | acc=0.809 #400 GT prediction | acc=0.772 #500 GT prediction | acc=0.851 #600 GT prediction | acc=0.765

In the below, I also visualize some bad samples where the prediction accuracy is lower than 0.5

#139 GT prediction | acc=0.305 #237 GT prediction | acc=0.456 #389 GT prediction | acc=0.300

Obviously, compared with the successful examples above, these failure samples contain not typoical "chair". For example, #237 is actually a wheelchair and #389 conatins two components apart. It reveals that the general category "chair" can in fact contain very diverse types of instacnes, which is a huge challenge to the generalizability of the model.

Q3

Ablation study over $num_points

I tried to adjust the parameter num_points during evaluation to see its influence. The results are as below.

• test accuracy from CLS model:

  • 10000 points: 0.940
  • 5000 points: 0.941
  • 1000 points: 0.940
  • 100 points: 0.891
  • 50 points: 0.813

• test accuracy from SEG mode:

  • 10000 points: 0.889
  • 5000 points: 0.884
  • 1000 points: 0.885
  • 100 points: 0.853
  • 50 points: 0.800

Ablation study over the ration matrix

I tried ti rotate the input points cloud with a certain angle wrt the XYZ axis. The results are shown as below.

• test accuracy from CLS model:

  • without rotation: 0.940
  • rotation by 10 degree: 0.939
  • rotation by 20 degree: 0.852
  • rotation by 60 degree: 0.268

• test accuracy from SEG model

  • without rotation: 0.889
  • rotation by 10 degree: 0.810
  • rotation by 20 degree: 0.717
  • rotation by 60 degree: 0.381