Clearer relationship between tileset and glTF metadata

[j9liu]

Tileset and glTF metadata are near-identical in schema/spec, yet they are functionally independent from each other. A tileset may have no metadata while its glTF contents contain metadata—and it's possible for each glTF to use different schemas. There is no sure way to deduce this information from a tileset.json.

However, there are some use cases where this creates friction with certain implementations. I've described two of them below.

Voxels

We override this behavior in the current version of 3DTILES_content_voxels, under the Class section:

The class property refers to a class ID in the root tileset schema. The class describes which properties exist in the voxel grid. [...] The class MUST match the class used to classify the glTF voxels in their EXT_structural_metadata extension.

This is because our current voxel rendering implementations require pre-allocating resources for a tileset's properties. These resources must be appropriately sized based on the TILESET_TILE_COUNT semantic and the types of metadata attributes. It helps that the extension explicitly forbids metadata differences among voxel tiles, but is this worth resolving against the base 3D Tiles spec?

Runtime Styling

Another case that exposes this inconsistency is metadata styling in Cesium for Unreal. CesiumJS can build a shader for every primitive it loads in a tileset, but Unreal cannot create shaders/materials at runtime. So in the Unreal Editor, we have to build a catch-all material that anticipates what metadata properties it will find to apply styling based on their values. To build that material, we have an "autofill" function that queries all tiles visible in the Editor viewport for their EXT_structural_metadata (and their EXT_mesh_features). But it's possible for properties of the same name to have different definitions in EXT_structural_metadata.

For instance, in OSM buildings there is a Year property that is null for some features. In an old implementation, this property was assigned a string type whenever null values were seen for the property (as a fallback). However, that resulted in Year having dual int and string definitions within the tileset. The metadata type that appeared for the user depended on which tile the algorithm found first.

(I think we may have patched this out, but it still illustrates the possible problems.)

Conclusion (or lack thereof)

These examples show how it is necessary (or at least helpful) to have tileset-level awareness of the metadata types within the glTF. It may be strong to argue for it as a requirement, but the lack of knowledge hurts some use cases.

I'm waffling on some of the potential avenues we could take. I don't think we should enforce one schema for the entire tileset, especially for tilesets with too many varying properties. Otherwise, if you wanted to grab a single model that only used one metadata property from the tileset, it would still depend on a very huge schema.

Plus, I think the current conception between glTF and 3D Tiles is that glTFs are the payload, and theoretically you should be able to render / handle a glTF independently of the tileset it comes from. But, that argument is hand-wavey if you think about factors like the coordinate system changes. Because 3D Tiles is z-up, the glTF contents are often rotated to cancel out the coordinate system change. But otherwise, the tiles are weirdly oriented in a solo context.

I don't know what a nice resolution to this problem would be, but it would be nice to figure out as we're thinking of #822. Thoughts and discussion welcome :)

[javagl]

(I only read over that once, but will probably have to digest it and draw connections to other issues/discussions/considerations)

A high-level point: It was intentional to have the same structure for both tilesets and glTFs. The idea of "metadata on all granularities" only really makes sense when this metadata has the same structure everywhere (as far as reasonably possible). There are some open questions and undesirable side effects to this. For example: Certain schema.json files have to exist in two places (specifications), to make each of these specifications self-contained. That duplication has... the effects that such a duplication usually has. For example, in another (long-ish) answer I pointed out the duplication of these classes in cesium-native, and there should preferably be a Cesium3DMetadata library to handle both tileset and glTF metadata.

The connection to (and specific point about) voxels is one of the things that I'll have to read more closely. But if I understood some other points correctly, then the following may be relevant:

• Both the tileset and the glTF could refer to an external schema, using the schemaUri, and this can be the same file
• The case that there are many glTFs with many different schemas does already raise a bunch of questions (and most of them are still open - one specific rabbit hole is that of qualified access)
• There might be a connection to https://github.com/CesiumGS/3d-tiles/issues/701 in some way: Maybe it should be possible to make an assertion, on the tileset level (!), that says: "I (the creator of the tileset) guarantee that all glTFs have a certain metadata structure". (Not sure whether that would help for the voxels case, though...)

[j9liu]

I knew you'd be first to respond @javagl! 😄

For example: Certain schema.json files have to exist in two places (specifications), to make each of these specifications self-contained. That duplication has... the effects that such a duplication usually has. For example, in https://github.com/CesiumGS/cesium-native/issues/760#issuecomment-1812785291 I pointed out the duplication of these classes in cesium-native, and there should preferably be a Cesium3DMetadata library to handle both tileset and glTF metadata.

Both the tileset and the glTF could refer to an external schema, using the schemaUri, and this can be the same file

Good point, I forgot to note that. The schema/class/classProperty definitions between both specs are almost identical, so it's not very satisfying to have all that duplication, and frankly I've wanted to implement that single metadata library for quite some time.

But if we had that single metadata library that worked for both specs... might it make sense, then, to unify definitions between both specs in the first place? I'm not advocating for this particular solution, but you could almost have a single and separate "Metadata Schema" spec that both 3D Tiles and glTF make use of. They can have different ways of storing / expressing the metadata, but through the same schema they have clear and shared expectations. Perhaps then it feels more legitimate to share the same schema.json file between a tileset and its glTF models.

The potential scale of a tileset is still an issue, especially that OSM Buildings example with 200+ properties. Well... I had a crazy idea in response to that which you're welcome to critique. What if a schema could contain external references to a class.json or something like that, where each class.json has a unique identifier:

{
  "classes": {
     "buildings": {
         "uri": "buildings.json"
      },
     "trees": {
         "uri": "trees.json"
     }
  }
}

That way a tileset could reference all the class.jsons used by its contents, but each content only references the class.json(s) that it uses. You could see if classes across a tileset resolve to the same URL -- and if so, properly treat them as the same thing. Maybe you could go even one step further to classProperty.json??? (Or maybe at that point I'm taking it too far 🤣 )

---

At risk of further complicating things, it's interesting to watch the conversation about glTFs of complex scenes (https://github.com/KhronosGroup/glTF/issues/2532) and how that might interface with 3D Tiles. What level of awareness should a complex glTF scene have about its external references, e.g., extensions used and their metadata?

To formally separate the issues I'm seeing, it's:

1. The duplication of specs between 3D Tiles / glTF, even though in some use cases they can (or should) be interlinked.
2. The parent / child relationships and their awareness of each other. Whether that's glTFs nested in glTF, or a tileset and its glTF contents, this question applies regardless of the spec(s) we're using.

(Sorry to add on to the pile of text to digest!)

[javagl]

might it make sense, then, to unify definitions between both specs in the first place?

Right now, I'm pretty sure that the files are (and if not: should be) just copy-pasted from one directory to the other. Any differences should just be technicalities, like some extends rootProperty vs. extends GltfProperty, with the latter being structurally equal again.

(The 3D Metadata Specification explicitly mentions 3D Tiles and the glTF extensions as "instances" of this specification, with the duplication of the schema.json files just being attempt to make them self-contained. There are some high/strict requirements when there are external references that should be normative)

What if a schema could contain external references to a class.json or something like that, where each class.json has a unique identifier:

This sounds very reasonable for me. There is another technicality - namely that there will be another level of "nesting" in that await readTileset(t) -> await readSchema(s) -> await readClass(c) chain. But that shouldn't block any structural improvement that could be made here. The idea of the schema (more) literally being a "class library" sounds like it could allow structuring complex schemas more cleanly.

There are some details to be thought through. Some may be related to https://github.com/CesiumGS/3d-tiles/issues/786. Others may revolve about the schema.id, which ... has somehow been neglected, given how important it may be in more complex schema structures and for longer-term evolvability.

(For example: When such a schema has the id: "example-schema-0.0.0", and refers to a exampleClass.json, and that class changes, then the schema ID may/will have to be updated...)