nerfacc
nerfacc copied to clipboard
nerfstudio-project
Obtaining visibility values
Hi, Is there any way to extract actual visibility values from the rendering process? Currently the visibility test returns a mask of visible / invisible samples, but what I want is to obtain the cumulative product of the alphas along a cast ray.
This doesn't seem to be calculated directly to obtain the weights.
Are you looking for the ray opacity?
https://github.com/KAIR-BAIR/nerfacc/blob/f9be5ad3b181058e27d484ac2e441973e797db18/nerfacc/vol_rendering.py#L100-L102
Not exactly, summing up the weights along the ray won't yield this directly. In the original nerf paper, equation (3) of C(r) states the color is the sum along the ray of two products: (1-exp(-xi)) and Ti. I am referring to Ti (which is itself a sum), roughly corresponding to how visible a point is "so far".
Oh I see that is actually a good feature request. Right now we don't support directly compute that but a hacky way is to compute the weights first, and then divided by alphas:
https://github.com/KAIR-BAIR/nerfacc/blob/f9be5ad3b181058e27d484ac2e441973e797db18/nerfacc/vol_rendering.py#L92-L94
https://github.com/KAIR-BAIR/nerfacc/blob/f9be5ad3b181058e27d484ac2e441973e797db18/nerfacc/ray_marching.py#L194
Note alphas can be zeros so you might want to use a eps when do division.
BUT I will support this in the next release!
If you accept pull requests I can push the edit I made that supports this. However, there seems to be a discrepancy between the calculation of alpha here: https://github.com/KAIR-BAIR/nerfacc/blob/f9be5ad3b181058e27d484ac2e441973e797db18/nerfacc/ray_marching.py#L194 vs here: https://github.com/KAIR-BAIR/nerfacc/blob/f9be5ad3b181058e27d484ac2e441973e797db18/examples/radiance_fields/mlp.py#L232 In the former, the official definition is used, but in the latter a 1st order Taylor approximation is used. Which one would you like to see in the calculation of Ti ?
Thanks for contributing! Did you make modifications in the underlying cuda? It would need both a forward and backward function. If so you are welcomed to open a PR!
Hi the former one should be used. The "opacity" defined in the mlp.py is only used for updating the occupancy grid, in which we only care about whether the opacity of a sample is larger than a small threshold, so 1st order taylor approx. is enough for the job. We use the official definition for the marching and rendering process!