nerfstudio icon indicating copy to clipboard operation
nerfstudio copied to clipboard

Affine transform from appearance embedding

Open AdamRashid96 opened this issue 2 years ago • 2 comments

Changes:

  • I added a flag --pipeline.model.use-appearance-embedding that defaults to True
  • When set to false the appearance embedding is not passed into the rgb mlp but is instead used to predict a scale and shift for the bottleneck vector. (Zip-Nerf)

Notes:

  • Results are variable
  • Overall seems to reduce floaters slightly and up the contrast of the scenes
  • Sometimes it can cause some extra noise
  • It also pushed up the ceiling in the Giannini Hall scene

Comparison: Video Screen Shot 2023-07-13 at 3 33 15 PM Screen Shot 2023-07-13 at 3 33 31 PM Screen Shot 2023-07-13 at 3 34 17 PM Screen Shot 2023-07-13 at 3 34 32 PM Screen Shot 2023-07-13 at 3 35 16 PM Screen Shot 2023-07-13 at 3 35 30 PM

AdamRashid96 avatar Jul 13 '23 22:07 AdamRashid96

Right now the behavior of using_appearance embedding is a bit confusing since when it's off, the appearance embedding is still used but it toggles between affine/nonaffine, right? It would be better to have the appearance embedding option expressed as a literal that toggles between 'off', 'affine', and 'mlp' or something

kerrj avatar Aug 17 '23 18:08 kerrj

Yeah I agree the wording is confusing and having two different toggles is unnecessary.

AdamRashid96 avatar Aug 17 '23 18:08 AdamRashid96