TensoRF
TensoRF copied to clipboard
apchenstu
Floaters in free space with limited input data
Hi Anpei,
I ran a modified version of TensoRF on my own dataset only with 12~20 images as input. The convergence and the quality are really nice but I see quite a lot of floaters in empty space (most seem to stick to the boundary of the bounding box). In addition, the boundary of the object is quite noisy in some novel views. I wonder if you have met with these problems before and if you have any suggestions to fix this problem? Thanks in advance!
could you post some results here? generally, TV regularization term would be helpful for real data
Sorry it is the data of someone's face so I couldn't show the full image. Here is the crop for the floaters (upperleft is the forehead, all others are floaters in the empty region):
This is trained with TV-L1 loss ( I want the boundary to be sharper), but still the amount of floaters doesn't seem to decrease. I apply a matting loss to ask regions outside the face to be complete empty. For me it is like some trash values in the blind spots of all cameras.
Hi, I had the same problem as you, did you solve it? It seems to be much better after I removed the background, is it a dataloader problem? （ I made my data in tankandtemple format, using the tankandtemple dataloader）
I implemented tensorf (without alpha mask / weight mask and with some design changes) in my own pipeline so I am not sure about the dataloader issue.
I didn't dig further into this problem since I have other bigger problems to solve right now. I might have some possible solutions. In the kiloNeRF paper, they seem to have similar issues. Their solution is to distill from the full NeRF model and add L2 regularization to the color network. Indeed my NeRF model doesn't have the floaters issue on the same data so I guess this might be a solution.
Please not that, the tensorf is only able to handle the bound scenes (without background) now, did you train with a background?
@apchenstu could you elaborate a bit more on why it cannot handle background? Thank you!!
@apchenstu could you elaborate a bit more on why it cannot handle background? Thank you!!
Basically, we didn't model the background, please refer to the limitation section in our paper.
Please not that, the tensorf is only able to handle the bound scenes (without background) now, did you train with a background?
Hi there, thanks for the explanation. But here is something that confuses me. How to define a thing is the background. Like the last examples in your supplement paper, It seems the tensoRF works fine on those real datasets, like Flower, Fern, Fortress, Horn, Leaves, Orchids, T-Rex, Room. Because Im doing my experiments based on tensoRF, the floaters stuff exists, and also the rendering quality of background appears not good, did I miss something?
Please not that, the tensorf is only able to handle the bound scenes (without background) now, did you train with a background?
Hi there, thanks for the explanation. But here is something that confuses me. How to define a thing is the background. Like the last examples in your supplement paper, It seems the tensoRF works fine on those real datasets, like Flower, Fern, Fortress, Horn, Leaves, Orchids, T-Rex, Room. Because Im doing my experiments based on tensoRF, the floaters stuff exists, and also the rendering quality of background appears not good, did I miss something?
It is indeed interesting to define the background of NeRF. Do you have any answers now?
