nerfstudio
nerfstudio copied to clipboard
nerfstudio-project
Support splat export in original dataset coordinates
The .ply export for Gaussians previously defaulted to saving in Nerfstudio's auto-oriented / auto-scaled coordinate frame.
I added support for exporting this in the original dataset coordinate frame, which the point cloud export supports via a "Save in world frame" checkbox.
I matched this in both the GUI and the export script CLI:
To me it makes the most sense to default this to True (I also flipped this for the point cloud export), but open to thoughts!
cc @jb-ye, #2909
I don't think the option would work actually for spherical harmonic parameters. The spherical harmonics are saved in the transformed frame and cannot be "re-orient" back to the original coordinates. (We can apply translation and scale, but not rotations, otherwise SH becomes inconsistent.)
My recommendation is to save the transforms as meta data as part of exporter.
I don't think the option would work actually for spherical harmonic parameters. The spherical harmonics are saved in the transformed frame and cannot be "re-orient" back to the original coordinates. (We can apply translation and scale, but not rotations, otherwise SH becomes inconsistent.)
My recommendation is to save the transforms as meta data as part of exporter.
Agree the current implementation is flawed, will revisit this after ECCV deadline!
I think ideally Nerfstudio just does a scale and re-center internally as part of the model rather than changing any of the camera poses, initial points etc that are part of the dataset.
Tough to overcome the inertia here but I agree that this would solve a lot of problems!
Tough to overcome the inertia here but I agree that this would solve a lot of problems!
Yes inertia but at least the PR discussions leave breadcrumbs. Things that fall off the rolling katamari ball become seeds for the next re-write.
Wait, spherical harmonics can easily be rotated by using Wigner matrices, right?
@jkulhanek Could you share a gist of sample code of rotating spherical harmonics? I am not aware of Wigner matrices.
Just a note here. Since SH is a complete basis, the rotation is possible exactly. Here is the code: https://github.com/jkulhanek/nerfbaselines/blob/develop/nerfbaselines/math_utils.py
The code is heavy, and I don't understand it fully, but I played with it in a notebook, and it seems to rotate the SH correctly. I can also share the notebook if you want.
It's the recursive impl (whatever it means) which is supposed to be more stable (for higher order). The good thing about it is that the wigner matrix can be computed once and then applied to all SHs at once.
Is there any update on this? I would also be interested in the ability to export splats in their original world frame. I haven't looked into it yet at all but would be willing to contribute.
Spherical harmonics functions are equivariant to SO(3) and can be rotated.
I don't know the exact code to do it, but probably possible with e3nn api:
From ChatGPT
import torch
from e3nn.o3 import Irrep, Irreps
from e3nn.o3 import spherical_harmonics, Rotation
R = torch.tensor([ [0.36, 0.48, -0.8], [-0.8, 0.60, 0], [0.48, 0.64, 0.6] ])
l = 2 # Degree of spherical harmonics
directions = torch.tensor([[0.0, 0.0, 1.0]]) # z-axis unit vector
Y_lm = spherical_harmonics(l, directions)
# Get the irreducible representation (Irrep) of the spherical harmonics of degree
l irrep = Irrep(f"{l}e") # e denotes even parity
# Apply the rotation to the spherical harmonics
Y_lm_rotated = irrep.D_from_matrix(R) @ Y_lm
Explanation:
- spherical_harmonics(l, directions): Computes the spherical harmonics of degree l for the given directions.
- Irrep(f"{l}e"): Creates an irreducible representation for the spherical harmonics of degree l. The "e" stands for even parity, which is typical for standard spherical harmonics.
- D_from_matrix(R): Computes the Wigner D-matrix for the rotation matrix R, which describes how the spherical harmonics transform under rotation.
- @: Performs matrix multiplication between the Wigner D-matrix and the original spherical harmonics to obtain the rotated spherical harmonics.
