**Assignment 1: Rendering Basics with PyTorch3D**

Student name: Kangle Deng

(##) Exploring loss functions
(###) Fitting a voxel grid

<table>
    <tr>
      <th>Optimized  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Groundtruth    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
    </tr>
    <tr>
      <td> <img src="./imgs/voxels_src.gif"> </td>
      <td> <img src="./imgs/voxels_tgt.gif"> </td>
    </tr>
</table>


(###) Fitting a point cloud

<table>
    <tr>
      <th>Optimized  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Groundtruth    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
    </tr>
    <tr>
      <td> <img src="./imgs/pointclouds_src.gif"> </td>
      <td> <img src="./imgs/pointclouds_tgt.gif"> </td>
    </tr>
</table>

(###) Fitting a mesh

<table>
    <tr>
      <th>Optimized  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Groundtruth    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
    </tr>
    <tr>
      <td> <img src="./imgs/mesh_src.gif"> </td>
      <td> <img src="./imgs/mesh_tgt.gif"> </td>
    </tr>
</table>

(##) Reconstructing 3D from single view

(###) Image to voxel grid

<table>
    <tr>
      <th>Input  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Predicted  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Groundtruth    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
    </tr>
    <tr>
      <td> <img src="./imgs/voxel/vox_0100_input.png" width="500"> </td>
      <td> <img src="./imgs/voxel/vox_0100.gif"> </td>
      <td> <img src="./imgs/voxel/vox_0100_gt.gif"> </td>
    </tr>
    <tr>
        <td> <img src="./imgs/voxel/vox_0300_input.png" width="500"> </td>
        <td> <img src="./imgs/voxel/vox_0300.gif"> </td>
        <td> <img src="./imgs/voxel/vox_0300_gt.gif"> </td>
      </tr>
      <tr>
        <td> <img src="./imgs/voxel/vox_0550_input.png" width="500"> </td>
        <td> <img src="./imgs/voxel/vox_0550.gif"> </td>
        <td> <img src="./imgs/voxel/vox_0550_gt.gif"> </td>
      </tr>
</table>

(###) Image to point cloud

<table>
    <tr>
      <th>Input  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Predicted  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Groundtruth    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
    </tr>
    <tr>
      <td> <img src="./imgs/point/point_0000_input.png" width="500"> </td>
      <td> <img src="./imgs/point/point_0000.gif"> </td>
      <td> <img src="./imgs/point/point_0000_gt.gif"> </td>
    </tr>
    <tr>
        <td> <img src="./imgs/point/point_0150_input.png" width="500"> </td>
        <td> <img src="./imgs/point/point_0150.gif"> </td>
        <td> <img src="./imgs/point/point_0150_gt.gif"> </td>
      </tr>
      <tr>
        <td> <img src="./imgs/point/point_0300_input.png" width="500"> </td>
        <td> <img src="./imgs/point/point_0300.gif"> </td>
        <td> <img src="./imgs/point/point_0300_gt.gif"> </td>
      </tr>
</table>

(###) Image to mesh

<table>
    <tr>
      <th>Input  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Predicted  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Groundtruth    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
    </tr>
    <tr>
      <td> <img src="./imgs/mesh_smooth01/mesh_0000_input.png" width="500"> </td>
      <td> <img src="./imgs/mesh_smooth01/mesh_0000.gif"> </td>
      <td> <img src="./imgs/mesh_smooth01/mesh_0000_gt.gif"> </td>
    </tr>
    <tr>
        <td> <img src="./imgs/mesh_smooth01/mesh_0150_input.png" width="500"> </td>
        <td> <img src="./imgs/mesh_smooth01/mesh_0150.gif"> </td>
        <td> <img src="./imgs/mesh_smooth01/mesh_0150_gt.gif"> </td>
      </tr>
      <tr>
        <td> <img src="./imgs/mesh_smooth01/mesh_0300_input.png" width="500"> </td>
        <td> <img src="./imgs/mesh_smooth01/mesh_0300.gif"> </td>
        <td> <img src="./imgs/mesh_smooth01/mesh_0300_gt.gif"> </td>
      </tr>
</table>

(###) Quantitative comparison

<table>
    <tr>
    <th></th>
      <th>Voxel   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Point Cloud    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
      <th>Mesh   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
    </tr>
    <tr>
      <td> F1 @ 0.05 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>
      <td> 74.600 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>
      <td> 93.360 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>
      <td> 88.330 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>
    </tr>
</table>

(###) Analyze effects of hyperparams variations

<table>
    <tr>
    <th></th>
      <th>w_smooth = 0.1   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>w_smooth = 1    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
      <th>w_smooth = 10   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
    </tr>
    <tr>
      <td> F1 @ 0.05 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>
      <td> 88.330 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>
      <td> 87.373 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>
      <td> 84.461 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>
    </tr>
</table>

<table>
    <tr>
      <th>Input  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>w_smooth = 0.1   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>w_smooth = 1    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
      <th>w_smooth = 10   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
      <th>Groundtruth    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
    </tr>
    <tr>
      <td> <img src="./imgs/mesh_smooth01/mesh_0000_input.png" width="500"> </td>
      <td> <img src="./imgs/mesh_smooth01/mesh_0000.gif"> </td>
      <td> <img src="./imgs/mesh_smooth1/mesh_0000.gif"> </td>
      <td> <img src="./imgs/mesh_smooth10/mesh_0000.gif"> </td>
      <td> <img src="./imgs/mesh_smooth01/mesh_0000_gt.gif"> </td>
    </tr>
    <tr>
        <td> <img src="./imgs/mesh_smooth01/mesh_0150_input.png" width="500"> </td>
        <td> <img src="./imgs/mesh_smooth01/mesh_0150.gif"> </td>
        <td> <img src="./imgs/mesh_smooth1/mesh_0150.gif"> </td>
        <td> <img src="./imgs/mesh_smooth10/mesh_0150.gif"> </td>
        <td> <img src="./imgs/mesh_smooth01/mesh_0150_gt.gif"> </td>
      </tr>
    <tr>
        <td> <img src="./imgs/mesh_smooth01/mesh_0300_input.png" width="500"> </td>
        <td> <img src="./imgs/mesh_smooth01/mesh_0300.gif"> </td>
        <td> <img src="./imgs/mesh_smooth1/mesh_0300.gif"> </td>
        <td> <img src="./imgs/mesh_smooth10/mesh_0300.gif"> </td>
        <td> <img src="./imgs/mesh_smooth01/mesh_0300_gt.gif"> </td>
    </tr>
</table>

We can see that increasing w_smooth will put more emphasis on the smoothness of the meshes.

(###) Inteprete your model

I visualize the probability of each voxel predicted by my model (blue means lower, and red means higher):

<table>
    <tr>
      <th>Input  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Predicted  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Probability  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
      <th>Groundtruth    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
    </tr>
    <tr>
      <td> <img src="./imgs/voxel/vox_0100_input.png" width="500"> </td>
      <td> <img src="./imgs/voxel/vox_0100.gif"> </td>
      <td> <img src="./imgs/voxel/vox2_0100.gif"> </td>
      <td> <img src="./imgs/voxel/vox_0100_gt.gif"> </td>
    </tr>
    <tr>
        <td> <img src="./imgs/voxel/vox_0300_input.png" width="500"> </td>
        <td> <img src="./imgs/voxel/vox_0300.gif"> </td>
        <td> <img src="./imgs/voxel/vox2_0300.gif"> </td>
        <td> <img src="./imgs/voxel/vox_0300_gt.gif"> </td>
      </tr>
      <tr>
        <td> <img src="./imgs/voxel/vox_0550_input.png" width="500"> </td>
        <td> <img src="./imgs/voxel/vox_0550.gif"> </td>
        <td> <img src="./imgs/voxel/vox2_0550.gif"> </td>
        <td> <img src="./imgs/voxel/vox_0550_gt.gif"> </td>
      </tr>
</table>

We can see the voxel model usually predicts lower values in the boundary voxels, and greater values in the interior area. This means the output of the model is continuous (smaller values when closer to boundaries), which makes it difficult to learn some hollow chairs with only several rods.

(##) Exploring some recent architectures

(###) Implicit Network

I implement the implicit decoder according to <a href="https://arxiv.org/pdf/1812.03828.pdf">Occupancy Networks: Learning 3D Reconstruction in Function Space</a>. The F1 @ 0.05 is 55.538.


<table>
  <tr>
    <th>Input  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
    <th>Predicted  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </th>
    <th>Groundtruth    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
  </tr>
  <tr>
    <td> <img src="./imgs/implicit/implicit_0000_input.png" width="500"> </td>
    <td> <img src="./imgs/implicit/implicit_0000.gif"> </td>
    <td> <img src="./imgs/implicit/implicit_0000_gt.gif"> </td>
  </tr>
  <tr>
      <td> <img src="./imgs/implicit/implicit_0100_input.png" width="500"> </td>
      <td> <img src="./imgs/implicit/implicit_0100.gif"> </td>
      <td> <img src="./imgs/implicit/implicit_0100_gt.gif"> </td>
    </tr>
    <tr>
      <td> <img src="./imgs/implicit/implicit_0650_input.png" width="500"> </td>
      <td> <img src="./imgs/implicit/implicit_0650.gif"> </td>
      <td> <img src="./imgs/implicit/implicit_0650_gt.gif"> </td>
    </tr>
</table>

<!-- (###) Parametric Network -->



<!--- Markdeep & image comparison library - probably no need to change anything below -->
<style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style><script src="./resources/markdeep.min.js"></script><script>window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")</script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
<script src="./resources/jquery.event.move.js"></script>
<script src="./resources/jquery.twentytwenty.js"></script>
<link href="./resources/offcanvas.css" rel="stylesheet">
<link href="./resources/twentytwenty.css" rel="stylesheet" type="text/css" />
<script>
$(window).load(function(){$(".twentytwenty-container").twentytwenty({default_offset_pct: 0.5});});
</script>