3D Tiles Next Metadata and XMP

[pjcozzi]

3D Tiles Next introduces a new metadata type system, encodings, and ability to make semantic extensions. See

• Introducing 3D Tiles Next, Streaming Geospatial to the Metaverse and
• Metadata Extensions.

This is a big step beyond 3D Tiles 1.0 that basically used the JSON type system and stored metadata in a Batch Table (texture).

In https://github.com/CesiumGS/3d-tiles/issues/519, we discussed:

3D Tiles has the opportunity to broadly solve spatial subdivision and metadata interoperability, e.g.,

... 3. glTF asset with just the metadata extensions to describe a car that can be imported into multiple metaverses 4. USD asset using a metadata semantic extension and the core metadata type system, but a USD-appropriate encoding

Including our interest in how this could work with glTF's XMP metadata extension, https://github.com/CesiumGS/3d-tiles/issues/519#issuecomment-949802921, CC @donmccurdy

I think this is a big enough topic that we should break it out into this separate discussion.

To bring 3D Tiles Next from an open specification to an open standard, I think we just need super clear guidance on when to use KHR_xmp_json_ld and when to use EXT_mesh_features, which I think we already have: asset-level vs. embedded-feature-level granularity.

Longer term, it would be great to know if and how 3D Tiles Metadata could leverage XMP, e.g.,

• Can XMP's RDF (Resource Description Framework) be used to define the new types that 3D Tiles needs (e.g., vector, matrix, and specific numeric types)?
• Can RDF be used to define new vocabularies for 3D Tiles Next?

I think this will require some initial insights from @weegeekps who knows XMP well, and lots of public exploration on how this could work and what the benefits are, e.g., less fragmentation in the ecosystem, path for clean glTF & USD metadata interop, etc.

Perhaps long term, the next generation of EXT_mesh_features is KHR_xmp_features.

I don't think we need all the answers right away, but just an idea of if we would want to go down this path and how it might work.

[pjcozzi]

Also see @donmccurdy's comments on this topic as well as potentially using Apache Arrow, https://github.com/CesiumGS/3d-tiles/issues/519#issuecomment-967640931

[ptrgags]

I found some old notes about a comparison of 3D Tiles Next metadata with XMP (more specifically the glTF KHR_xmp_json_ld extension. Adding it here for reference.

	3D Tiles Next	glTF + XMP + JSON Linked Data (JSON-LD)
Encoding	Binary for fine-grain details (per-vertex, per-pixel), JSON for coarse-grain details (tileset, group).	JSON only via JSON-LD
Columnar format?	Yes	No
Granularity	Tileset, tile, group, feature (per-vertex to per-primitive), per-texel	Attached to arbitrary parts of a glTF file. No per-vertex or per-texel metadata
Useful for	Efficiently packing metadata for runtime use (UI, styling, analysis)	Describing meatadata about documents (in this case a glTF asset), attaching metadata to different parts of the glTF
Schemas	Can be either embedded or external	Embeded or external via JSON-LD
Numeric Types	explicitly-sized types (UINT8, INT32, FLOAT64), etc.	Integer and Real (stored in JSON, so on disk it's a string)
String types	STRING. specializations could possibly be implemented using semantics	Text with several specializations such as URI, Date or MIMEType
Array types	ARRAY (both fixed and variable length)	Arrays and Sets (variable length), implemented via JSON-LD
Enum types	ENUM - integers or strings	Choice - string-based enums, though slightly different (e.g. they allow open-ended enums)
Existing Standards	N/A	XMP: ISO 16684-1, XMP + JSON-LD: ISO 16684-3
Existing Namespaces	N/A	Many, most notably Dublin Core

[weegeekps]

I finally have some (limited) time to sit down and write up some of my notes as I've been reviewing and thinking about EXT_mesh_features. Overall, I really think the direction and implementation looks great and as per our past conversation I am looking at ways that we can utilize XMP to help us here. That exercise has raised more questions than answers at the moment:

• XMP (and JSON-LD) typically don't have explicitly sized types, so we would need to find and define our own definitions for these. These would have to be implementation notes for anyone implementing a parser using Features with XMP. I question if that's acceptable or if it's an unnecessary sharp edge. Definitely worth discussing more.
• I am unsure that XMP is compatible with the efficient storage of data via the Property Tables. Today the KHR_xmp_json_ld extension relies on packets being stored within the glTF JSON. In theory, I don't see why we couldn't store packets in a glTF buffer view, but we would need to store the entire packet, which would be considerably larger size than the existing property tables. More discovery is needed here.
  • How do we want to separate out what goes into an XMP packet, and what goes into a Property Table? Can entries within a property table reference XMP packets as one of the columns?
  • What does the overhead of storing an XMP packet in a buffer view look like? Is it even worth doing so?

At this point, I imagine an ideal solution will be one that relies heavily on the Property Tables to store lots of small pieces of metadata, where XMP packets are used for storing more detailed shared metadata between multiple features. Although I think there might be merit to storing XMP packets into a buffer view, and then parsing them all at runtime. The overhead may be far smaller than what I'm currently suspecting. Regardless it's worth looking at closer.