Multisampling realisation

[ilbash]

Hello, @SuLvXiangXin ! Thank you for your work. I see your multisampling code and I have some questions.

1. What is cam_dirs? I see this defenition on code but it gives me zero ideas why did you use cam_dirs in multisampling.
2. If we want to constract two vectors perpendicular to the view direction, we can use direction vector instead of cam_dirs, e.g.

# two basis in parallel to the image plane
rand_vec = torch.randn_like(directions)
ortho1 = F.normalize(torch.cross(directions, rand_vec, dim=-1), dim=-1)
ortho2 = F.normalize(torch.cross(directions, ortho1, dim=-1), dim=-1)

Please correct me if I'm not right. Thanks!

[SuLvXiangXin]

@Ilyabasharov You can have a look at the MipNeRF paper. They truncate the ray cone parallel to the image plane, rather than perpendicular to the view direction. I think ZipNeRF may adopt this configuration.

[ilbash]

@SuLvXiangXin do you mean this? 4 page of MipNeRF paper

[SuLvXiangXin]

@Ilyabasharov You can see that the circle in the ray cone is parallel to the image plane, and they actually do like this in their implementation https://github.com/google-research/multinerf

[ilbash]

In Zip NeRF authors said that So in your code base_matrix should be orthonormal. But first two vectors are perpendicular to cam_dirs and the third is directions. Question: cam_dirs must be parallel to directions, doesnt it?

[SuLvXiangXin]

Yeah, in the very first commit, I truely implement like that. But there comes some problems, how can I compute $\dot{r}$? While it is defined on the image plane, if I use directions as cam_dirs, that is not correct. Another solution is to use $\frac{\dot{r}}{norm(directions)}$ instead of $\dot{r}$, but this is weird.

[ilbash]

I think all is ok in realization with directions because you construct orthonormal basis (length of each vector is equal 1) and than you multiply it to radius value defined and half of pixel area. otherwise, there will be no orthonormal basis with one of the vectors equal to directions. $\dot{r}$ is a scalar and can be computed as here.

[ilbash]

@SuLvXiangXin updated figure from the paper

Because of this, I'm leaning towards this implementation

# two basis in parallel to the image plane
rand_vec = torch.randn_like(directions)
ortho1 = F.normalize(torch.cross(directions, rand_vec, dim=-1), dim=-1)
ortho2 = F.normalize(torch.cross(directions, ortho1, dim=-1), dim=-1)

[SuLvXiangXin]

@Ilyabasharov Wow, thanks for reminder, I'll checkout the updated version of the paper.