KhronosGroup
KHR_gaussian_splatting
Update: I've split out KHR_gaussian_splatting_spz_2 into it's own PR. There's been many changes since I wrote the summary, and while I'm leaving it at the moment. It will be updated soon.
This extension proposal, KHR_spz_gaussian_splats_compression, allows for efficient storage of 3D Gaussian splats data within glTF using the SPZ compression library from Niantic Spatial. The extension is applied to a primitive. The SPZ binary blob is stored as a buffer within the glTF file, and implementations can use the SPZ library to either decompress and then map the compressed Gaussians into placeholder attributes on the primitive or directly decompress into their rendering pipeline if preferred. Content creators have the flexibility to choose to use no Spherical Harmonics, or up to all 3 degrees of spherical harmonics depending on their use case.
We are currently working on an implementation in the CesiumJS engine based on this draft that we hope to have released soon.
It appears that this extension is tied to the Niantic Spatial library. Is it necessary to specify a version or other unique identifier to ensure that the desired algorithm and calling sequence is used?
Also note that the license for the Niantic Spatial library is MIT.
Is it necessary to specify a version or other unique identifier to ensure that the desired algorithm and calling sequence is used?
Good question. The SPZ library packs along a version number within the binary data that we store in the buffer, so it's unnecessary to have a version number stored in the glTF metadata.
The SPZ library packs along a version number within the binary data that we store in the buffer, so it's unnecessary to have a version number stored in the glTF metadata.
Devil's advocate: Could a version information in the glTF be necessary exactly because of that?
The SPZ format version currently seems to be "2". Will this also work if the version of the SPZ data stream is "1" or "3"? (Likely yes - there are not too many degrees of freedom for what splats consist of: Position/Color/Scale/Rotation/SH's. If the differences between version numbers of SPZ are only how these elements are represented in the binary form, then the version does not matter (assuming that the data can always be converted to or interpreted as VECn/GL_FLOAT data). If version 3 could omit the 'rotation' (or suddenly represent that with Euler angles 🤪 ) then this could cause a problem.
This is an excellent point, @javagl. As of this moment, the SPZ library's loadSpz function returns a GaussianCloud type which does not contain the version information, but that information could be useful for an implementation in the event that the shape of the GaussianCloud type changes. More importantly, I can see where we would need to know this before ever reading the SPZ blob.
With this being tied so closely to the SPZ implementation, the crucial question is exactly that: Can The GaussianCloud type change? (And if so: where and how is this communicated).
If this type is guaranteed to never change, then the version information may not be required in the glTF. The internal, binary representation of the SPZ may change, but that can be handled internally with the version field in the header, and clients don't have to care about that. They still receive the same GaussianCloud(Data?) object as before. But even then, there may be further questions. What happens when you have an "old" version of the library (that only supports version=2), and suddenly receive a new SPZ data stream that contains version=3? Does it crash, return a fallback, return undefined...?
A POSITION attribute accessor MUST have min and max defined (as of https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#accessors-bounds ). The accessors section currently says that the position accessor should contain max: [1, 1, 1], min: [-1, -1, -1 ]. I'm not sure how to straighten that out. Could it make sense to require these values to match the actual minimum/maximum that are found in the splat positions? (Computing these values for an accessor without a bufferView may be tricky regardless, but one could argue that this is left to the implementor/writer...)
EDIT: The inlined sample declares material: 0. To my understanding, there cannot be a material associated with that, right?
The accessors section currently says that the position accessor should contain max: [1, 1, 1], min: [-1, -1, -1 ]
I don't think our intention was to state that the min and max values should be all -1 and 1 respectively. I will fix this.
We've gotten feedback about this extension regarding concerns with SPZ's precision in some circumstances. It's been found that in certain cases, the error introduced by the quantization of the quaternion can be significant. When the error occurs on a thin body, it can be visible, but in our own testing this seems to be a rare case. We're wondering if the community, especially those who've used SPZ themselves, have anything they'd like to add regarding their own investigations.
When we initially investigated SPZ as a compression format, we compared PSNR values of between uncompressed splats, meshopt with "high" settings (16 bits for position, rotation, and scale), and SPZ. It seemed that SPZ offered superior compression while keeping the PSNR at an acceptable level. Generally, depending on the dataset, our "high" settings meshopt did score better at ~51 dB compared to ~46 db, but with 25% larger file sizes.
While we do think that the error SPZ introduces into rotations can be significantly reduced by just moving from 1-byte to 2-bytes to store the rotations, we are interested in understanding the community's perspective and ideas for this issue. Moving from 1-byte to 2-byte within SPZ isn't too expensive. For one of our larger datasets (522.8 MB), we have gone from 37 MB (92.9% compression) to 42 MB (91.7% compression). The impact seems fairly linear with overall source filesize.
@javagl, it'd be valuable to hear your take on this.
The disclosure again: I'm an independent contributor for the Khronos Group. But I'm also a freelancer who happens to work for Cesium right now. I'm NOT actively involved in the development of this extension or its support in Cesium products. I'm NOT speaking on behalf of either side here. I neither actively endorse nor oppose this extension proposal.
It's difficult (for me) to translate a PSNR value into actual, perceived visual quality. This is not about a single 2D image where it's largely about 'how grainy the image looks', but about a highly complex "3D object" (and, in doubt, how exactly it is rendered). I know that there are other quality measures, e.g. the ones mentioned in https://w-m.github.io/3dgs-compression-survey/ , but I won't claim to understand them, and even less what the "threshold" for any of these values is at which small features (like power lines) turn into visually unpleasant noise. The difficulty of using these measures for the comparison of different approaches also lies in the fact that some compression methods offer a configurable degree for the compression, which is usually a direct trade-off with 'quality' (and that trade-off does not necessarily have to be linear)
There seems to be a consideration to change the quantization size for rotations in SPZ. This is related to the points mentioned above, about versioning. Those who are only using the SPZ library to load the data may not be affected by that, iff the API of the library does not change. But those who implemented their own SPZ tooling will have to handle this change. And whether or not there may be further changes (maybe even changes that affect the API), and how that could be communicated via the extension, is not yet clear.
Version 1 of SPZ used 16 bits for the positions, which was extended to 24 bits in Version 2. Version 2 of SPZ uses 8 bits for the rotations, which is now considered to be extended to 16 bits.
The fact that these changes imply a (major) version change could cause some scrutiny. But there certainly is a trade-off: Right now, the format is really simple. If it was supposed to be more generic and "resilient" for such changes, then it would be necessary to make each and every aspect (i.e. all the bit counts) configurable, increasing the implementation effort.
In the best case, these versions can be seen as iterations that have been necessary to find the "sweet spot".
On the most general level:
I think that it would be nice if we could use the existing compression infrastructure, where the required accessors are stored in plain buffer views, to which any (arbitrary, existing) compression technique (including quantization) could be applied. There seem to be reasons why this is not considered to be a viable path. (Although I don't know the exact reasons...).
Beyond that, most of the things that I could say about SPZ or its use in the extension would have an overlap to what I already said in the main glTF Splats discussion issue, or have already been mentioned in the working group calls.
- On the one hand, it's not nice to tie splats in glTF to a specific format and library. On the other hand, the format is simple and the library is open source
- On the one hand, it's not very sophisticated to just "inline" raw file data into a buffer view. On the other hand, regardless of what the format will be, it will be used for filling a bunch of
POSITION/_SCALE/_ROTATION...accessors - On the one hand, the format is not much more than a ZIPped memory dump, and it's likely that some format will be "better" in one way or another. On the other hand, it's exactly that simplicity that could help a quick adaption. (Anyone up for implementing a Draco decoder....?)
A low-level remark about the SPZ format itself: It is inherently not suited for any form of streaming. The layout as a "structure of arrays" makes it impossible to load and display the data progressively. Is that an issue? I don't know. The fact that it didn't come up until now might indicate that this is not important.
It's difficult (for me) to translate a PSNR value into actual, perceived visual quality.
This is a fair point, but PSNR is largely still used in 3D comparisons because of its overall reliability in predicting what is generally a subjective point of view, "how good does compressed item look compared to uncompressed?" Generally, an image (3D or 2D) with a PSNR of 99 dB is considered, "perfect." In practice, it's rare, if not impossible, for any image to achieve this score in a proper testbed, and in my own experience with PSNR I've never seen lossless formats actually achieve anywhere close to that value. In scientific studies, a value above 50 dB is generally considered as perceptibly equal to a reference image.
In my testing for each compression (uncompressed, SPZ, meshopt), I took screen captures at many different angles and used those to generate PSNR values for each angle. I then took those values and applied the arithmetic mean to get an average PSNR. Uncompressed and SPZ were easy with no additional settings, but meshopt we actually tested 3 different sets of settings, with "High" being 2-bytes for position, rotation, and scale, "Medium" being 2-bytes for position and 1-byte for rotation and scale, and "Low" being 10-bits for position and 1-byte for rotation and scale.
As mentioned above, meshopt performed the best, scoring an average of 51 dB (range: 49 dB to 53 dB), but at 25% of additional cost. Our "medium" configuration for meshopt scored far worse, at an average of 41 dB (range: 39 dB to 42 dB). In comparison SPZ scored an average of 46 dB (range: 45 dB to 46 dB). SPZ's consistency, close vicinity to 50 dB and superior file size is largely why we chose it for our current iteration of this extension.
I have some bitmaps that I can share with you if you're interested in seeing the visual difference. I'm unable to tell a difference between uncompressed and SPZ but can definitely see a difference with our "medium" meshopt profile. GitHub adds its own compression to the bitmaps and makes them blurry eliminating the differences, so I am hesitant to post them here as they will be misleading.
In the best case, these versions can be seen as iterations that have been necessary to find the "sweet spot".
This is exactly the mindset we've been taking on this. SPZ isn't perfect, but it's easy to implement anywhere. There will be changes over time as Niantic and others improve and mature the compression. Today, it does support versioning and the way we've set up this extension will allow for supporting newer versions without a need to update the extension specification.
think that it would be nice if we could use the existing compression infrastructure, where the required accessors are stored in plain buffer views, to which any (arbitrary, existing) compression technique (including quantization) could be applied. There seem to be reasons why this is not considered to be a viable path. (Although I don't know the exact reasons...).
The reason was simplicity and reducing fragmentation, but as I've commented over in the other issue this is worth revisiting.
There will be changes over time as Niantic and others improve and mature the compression.
It may sound stubborn, but: Is that really necessary? At least, I usually appreciate some form of reliability and stability when it comes to data formats. When I'm creating a file today, then I want to be able to open this in a viewer two years from now. And when I'm writing a viewer with support for glTF+SPZ, then I want to be sure that it can open a file that is created two years from now. The point is: If SPZ (in its current form, or v3 with the two-bytes-rotation) really is that "sweet spot" that covers 95% of all use cases satisfactory, then there should hardly be a reason for new versions.
Of course, if there was a huge, justified change, like a sudden size reduction by 50%, or a fundamental change in the set of supported attributes, then this could warrant a new version. But for the sake of the stability of glTF and its ecosystem, one could then consider make this explicit, with a new KHR_spz_latest_and_greatest_gaussian_splats_compression extension. If the SPZ format evolved into v4, v5, v6, each just squeezing out a few percent of compression or adding/removing some quantization bit here and there, then this could feel like a "moving target"...
The point is: If SPZ (in its current form, or v3 with the two-bytes-rotation) really is that "sweet spot" that covers 95% of all use cases satisfactory, then there should hardly be a reason for new versions.
I think we are mostly aligned here. Improvements to SPZ should be rare, especially if SPZ really has hit that sweet spot of 95% of cases. In the event that a change is required, my understanding is that the desire is to allow the library to handle old versions. I think this is ideal. If a v3 is created, the library should still support v2 files just as the library currently still supports v1 files.
Requiring that viewers check the version of the SPZ prior to opening it doesn't seem like an excessive ask to me. If the viewer hasn't updated to the latest version, they fail with a message alerting the user to the issue. Especially considering your excellent point to include the version with the extension. Requiring new extensions for new versions of SPZ seems unreasonable to me. This extension ultimately just stores the SPZ blob and any necessary metadata required to use that blob, similar to how the Draco extension stores the Draco bitstreams.
@weegeekps
A KHR extension must have a fixed scope because of IP issues, i.e., it cannot support "SPZv3 and later" because no one knows what IP could be contained in the future versions. On top of that, it would be against glTF portability and compatibility guarantees to imply any specific relations between SPZ bitstream formats and the reference software.
Both excellent points, @lexaknyazev.
This extension may potentially have some small changes soon based on the feedback here:
- We're considering breaking out the core "3D Gaussian splatting" portion into its own extension. This extension would then only contain the SPZ compression components.
- We will lock down this extension to a specific version of SPZ, to be decided pending some ongoing discussions.
I took a first pass at splitting out the extension into a base extension and a SPZ extension. It's a rough draft but it should be a solid starting point for now. I did not just resurrect our original extension from late last year named KHR_gaussian_splatting and instead rewrote it based on the KHR_spz_gaussian_splats_compression draft.
Please let me know your thoughts and tweaks you'd like to see.
P.S. The min and max values in the accessors section is not fixed yet. It will be soon.
We developed a tool to convert a ply or splat file to KHR_spz_gaussian_splats_compression format 3dtiles. it is in our product GISBox and it is totally free to use. you guys can use it to test this new feature.
I just made a small change to the SPZ compression extension to (hopefully) simplify the language around inheritance from the base KHR_gaussian_splatting extension.
@lexaknyazev very interested in any thoughts you may have with this language.
I think that the KHR_gaussian_splatting extension should inculde a resolution for https://github.com/KhronosGroup/glTF/issues/2111. Athough that issue is still open an we have not explicitly agreed on "THE" solution that will be universally applied, I think that "namespacing" by prefixing the attribute names with KHR_gaussian_splatting sounds like a reasonable first shot. Even if there will be a different solution in the future, we could still have a solution for the current extension that is "safe", insofar that people who generate and read this data don't have to adjust their writer/reader code.
In the current version of the proposal, the accessors for these attribute do not define a Obsolete based on https://github.com/KhronosGroup/glTF/pull/2490#discussion_r2225871885byteOffset. This implies that the attributes have to be assumed to be stored in the buffer view in the order in which they are mentioned in the specs. I think that this should be clarified.
I think that the KHR_gaussian_splatting extension should inculde a resolution for https://github.com/KhronosGroup/glTF/issues/2111.
I agree. I made a comment over in #2111 about a potential solution.
Regarding the byteOffset, do we need to explicitly specify that this extension supports them? Wouldn't this be covered by the base glTF specification since we don't explicitly specify that we don't support byte offsets?
I've been following what the OpenUSD folks are doing as well as talking with a few of them, and below are some proposed changes that I want to run by everyone before I make them to the specification. Some of these suggestions have been mentioned prior in other conversations, but I wanted to expand on them here and go into a bit more details now that I have a better understanding of them.
All of these additions are in the spirit of keeping this base extension forward looking.
Shape property
Researchers are currently looking a range of shapes for Gaussian splats. Today, we've seen research around the traditional ellipsoid shape, triangles, and quads. Adopting a property to specify what shape the Gaussians would be good for future-proofing.
Similar to what OpenUSD is doing, in the spec we would provide an initial list of some known options: ellipsoid, triangle, and quad. The default value will be ellipsoid. The property will be optional and omitting it will indicate the default value should be used.
We will likely need a mechanism for allowing custom values here. OpenUSD often is okay with stringly-typed enumerations, but within glTF we tend to be a bit more conservative.
Rendering Hints
The most significant differences between the Gaussian Splatting API that the OpenUSD folks are putting together and KHR_gaussian_splatting are best categorized as rendering hints. These are values that are known during training that must be passed along to the renderer to allow for proper rendering.
All of these hints would live within a hints object within the extension definition.
Like the shape property, it's impossible for us to know all of the possible future values for these. As hints, we'll define a "default" for each of these, and then explicitly state that implementers should fall back to the default if they come across a value they don't recognize, and attempt best effort at rendering.
It's also up to the renderer themselves if they use these hints for their renderers or not and use of them will be explicitly optional. This is in the same spirit as what OpenUSD is doing with these: the goal is to provide information that is useful to renderers but not make any assumptions as to whether or not those renderers use them.
Sorting Method
The sortingMethod specifies how the splats are sorted for the rendering process. The initial possible values match what OpenUSD is doing:
-
cameraDistancewhich is the distance between the center of the splat and the position of the camera. -
zDepthwhich is the projected z-depth in the camera projection.
The default value will be cameraDistance.
Projection
The projection specifies how the splat shape is projected onto the image. Initial values diverge slightly from what OpenUSD is doing:
-
perspectiveis your typical 3D perspective projection based on scene depth. -
orthographicprojects the splat orthogonally into the image. OpenUSD refers to this astangentialtoday.
The default value will be perspective.
Sample with Shape and Rendering Hints
All of the fields proposed above are optional, but this is what it would look like with all of them defined.
"meshes": [{
"primitives": [{
"attributes": {
"POSITION": 0,
"COLOR_0": 1,
"_SCALE": 2,
"_ROTATION": 3,
"_SH_DEGREE_1_COEF_0": 4,
"_SH_DEGREE_1_COEF_1": 5,
"_SH_DEGREE_1_COEF_2": 6
},
"mode": 0,
"indices": 7,
"extensions": {
"KHR_gaussian_splatting": {
"shape": "ellipsoid",
"hints": {
"sortingMethod": "cameraDistance",
"projection": "perspective"
}
}
}
}]
}],
Including Spherical Harmonics in KHR_gaussian_splatting
With the Gaussian Splatting API that the OpenUSD maintainers are working on, they are splitting out Spherical Harmonics into a separate API. The thought process here is that in the future there may be no need for Spherical Harmonics, and they'd have another API they'd define later with those properties. Within our team here at Cesium we went back and forth on this trying to decide if we'd follow suit and we've decided against doing so.
Currently, the Spherical Harmonic data in KHR_gaussian_splatting is optional. Ideally, when PBR materials for 3D Gaussian splatting become possible in a wide variety of renderers, we will use the existing mechanisms for materials within glTF and the Spherical Harmonic data will simply be omitted. If there are special properties for those materials, a material extension can be defined later to provide those on the material object. Given the simplicity of this approach, we don't currently see a need within glTF for splitting Spherical Harmonics out into a separate extension.
Thanks for the updates @weegeekps and taking inspiration from OpenUSD gsplats for aligning formalization discussion.
@willeastcott & @slimbuck Following the impressive work you handled around Self-Organizing Gaussians for rendering and conversion via playcanvas/splat-transform, would @playcanvas be interested in formalizing a glTF extension like KHR_sog_gaussian_splats_compression extending KHR_gaussian_splatting?
This could help formalize a standard specification for storing, consuming SOG-compressed gsplats for broader adoption - for example also within other renderers.
This could help formalize a standard specification for storing, consuming SOG-compressed gsplats for broader adoption
I think that this is the crux for even trying to extract a "baseline" specification like KHR_gaussian_splatting. I've been poking several people, inquiring whether the attributes that are currently defined in that baseline specification are suitable for something that could be called a "general splat support for glTF".
One could argue that such a baseline specification would be useless if KHR_spz_gaussian_splats_compression was the only specification that could be built on top of it. That's not entirely true, though: The baseline specification already does allow the splat data to be stored in good old plain accessors, in uncompressed form. And it already does allow to apply some of the existing compression machinery (like 'meshopt') on top of that. This means that there already is something that a renderer could rely on, and that producers could generate with relatively little effort, including a compressed form of splats.
But it could still be important to make sure that this structure (i.e. the attributes that are defined there) could be "filled" or "fed" with data that is compressed in other ways. If it is possible to fill that baseline spec with life, using both SPZ and SOG, then this is a strong hint that it could also be used for a third or fourth compression method that may be developed... tomorrow.
Caveats:
The use of that base extension:
As far as I understood, it is by no means clear how splat data is supposed to be sent to a renderer. Some renderers might try to keep it simple, and use VBOs (although I've heard that this may not be the best choice). Some shaders (e.g one that I took from a Python renderer and used in a Java renderer) are using SSBO (Shader Storage Buffer Objects). I don't know whether this is good or bad, but ... it was relatively simple, and it seems to work reasonably well. Others are taking the SPZ data and are encoding that into textures in some way. (It looks quirky, but ... usually, when something like that looks "quirky", people argue with "performance" (and usually, when people argue with "performance", there are no benchmarks... however)). I think that for SOGs, the data is already stored in textures to begin with, and it seems reasonable to assume that these textures are supposed to be sent to the shader directly.
So if someone wants to implement splat support for glTF in a renderer, then it should at least be possible to write a renderer that just consumes the attributes that are defined in the baseline specification. This may not always be the "best" choice, and it may have performance drawbacks for the rendering itself, or require forms of transcoding. But the renderer (or engine) still has the option to include deeper, more specific knowledge about the actual extension, and how the data is actually stored. The renderer could look for the SPZ version, and use that 'encoded-textures' approach. The renderer could look for the SOG version, and used these textures directly. If none of that is found, (maybe because the splats are just stored as plain accessors+meshopt, or another compression method), then it could still render them, and wouldn't have to bail out just because it doesn't know the latest-and-greatest "compressed splat rendering" method that someone came up with.
The limits of that base extension:
There will be animated splats. We know that. Will there be "morph targets" for splats? Maybe. Will there be additional attributes for splats? Probably, in some form. Can the additional information that is required for this "next-generation splats" be encoded in that baseline extension? No! And that is not the goal. When people agree on what "animated splats" are, and what attributes they need, then there can be a KHR_animated_gaussian_splatting extension that defines the baseline structure, and where different compression methods can fill that with life.
(All that may sound naive for someone who's more deeply involved in splat (rendering) research. I'm just trying to ensure that it makes sense to establish such a baseline specification. Feel free to chime in with additional thoughts about where this does or does not make sense)
Pushed some changes to the SPZ extension. Still working on updates to the base extension.
Pushed changes to the base 3DGS extension:
- Updated the extension based on feedback.
- Included the shape and rendering hints mentioned in my comment above.
The title and description of the PR should have KHR_spz_gaussian_splats_compression changed to KHR_gaussian_splatting_spz_compression to match the extension name.
@weegeekps Will add the feedback from my 3D Formats presentation here as well.
COLOR_0 could be considered to keep its original semantic.
One option would be, that the splat is colorized afterwards, especially using the alpha channel to e.g. fade in and fade out the complete splat cloud.
To keep the mentioned fallback, this value could also point to the same accessor as for the later described spherical harmonics extension with degree 0 and coefficient 0 (this I did not mention during the 3D Formats call and did come to my mind the last days during the ongoing discussions).
In regards of DEGREE_x_COEF_y, suggest to create a separate extension for spherical harmonics (SH). As SH are quite often used in computer graphics, having a separate extension avoids a potential overlap in the future and it could be reused for upcoming use cases.
So, DEGREE_0_COEF_0 would be included in this case and please also compare with the COLOR_0 comment.
Also suggest to have a separate extension for the transformations for later reusage:
TRANSLATION
ROTATION
SCALE
Also, please note of the existing and mentioned https://github.com/KhronosGroup/glTF/tree/main/extensions/2.0/Vendor/EXT_mesh_gpu_instancing extension during 3D Formats call.
Difference is, that the suggested new one is independent from the implementation (Usage of GPU instancing) and maybe we can combine them somehow (e.g. put the GPU instancing in the non-normative section).
The KHR_gaussian_splatting extension would then mainly depend on the previously described glTF extensions.