Assignment 5

Linghan Xu
linghanx@andrew.cmu.edu

Q1. Classification Model (40 points)

Best test accuracy: 0.9748

Success prediction:

Chair:

ccc

 

Vase:

ccc

 

Lamp:

ccc

 

Failure prediction:

Ground truth: Chair & prediction: Lamp

c

 

Ground truth: Vase & prediction: Lamp

c

 

Ground truth: Lamp & prediction: Vase

c

 

For the first failure prediction, I think it's because the chair is folded and its shape is different from most chairs in the dataset.

For the second and third failure predictions, I think it's because some vases and lamps are similar to each other in shape.

Q2. Segmentation Model (40 points)

Best test accuracy: 0.9059

Visualization:

Acc: 0.9435

cc

Acc: 0.9913

cc

Acc: 0.9656

cc

Acc: 0.5144

cc

Acc: 0.5648

cc

I think it's because for the good predictions, samples have clear boundaries between different semantic parts, but for the bad ones, the semantic parts aren't so clear, such as the yellow part and blue part, which are difficult to predict even for humans.

Q3. Robustness Analysis (20 points)

  1. Rotate the input point clouds by 30 degrees and 60 degrees along the z-axis, respectively.

    1. For classification, rotating 30 degrees makes the test accuracy to drop from 0.9748 to 0.7975. Rotating 60 degrees makes the test accuracy to drop to 0.3127.
    1. For segmentation, rotating 30 degrees makes the test accuracy to drop from 0.9059 to 0.7373. Rotating 60 degrees makes the test accuracy to drop to 0.5891.

  2. Input randomly selected 1000 points and 100 points per object, respectively.

    1. For classification, inputing 1000 random points per object doesn't affect the test accuracy (from 0.9748 to 0.9696). Inputing 100 random points makes the test accuracy drop to 0.8646.
    2. For segmentation, inputing 1000 random points per object also doesn't affect the test accuracy (from 0.9059 to 0.9018). Inputing 100 random points makes the accuracy drop to 0.8178.