EXT_property_animation extension proposal

[najadojo]

[najadojo]

Would you help me out by describing what types of properties would need recompilation? I've attempted to avoid this particular issue by limiting the data types and not allowing index object references.

I'm not apposed to having this extension enumerate all the properties that can animate including those in the current accepted and draft extensions. Going forward extensions could reference this spec and indicate which properties are allowed. If there is a way to avoid this level of detail I think we'd benefit from it.

[donmccurdy]

A few likely-to-be-problematic examples that would seem to be allowed as written:

• accessors.byteOffset, .min, .max, .count
• accessors.sparse.count
• bufferView.byteStride
• buffer.byteLength
• extras.*
• material.*.texCoord
• node.matrix
• much of BLENDER_physics
• KHR_techniques_webgl uniform.count and uniform default values
• KHR_texture_transform offset/scale/rotation
  • This example is one I would like to see work, but (AFAIK) won't in three.js today

This feels like enough examples that new extensions are commonly going to run into special cases. What about a whitelist of properties in the core spec, and a provision that current and future extensions may "opt in" with their own whitelist? It does not necessarily need to be an exhaustively maintained whitelist. For example, material/* excluding texCoord would cover every case I care about in the core spec.

[donmccurdy]

I had this initially but the engines I looked at need to collect and clone the materials as if the path implies the channel should only effect the single mesh primitive and not others that use the same material reference.

Engines already have to solve that; e.g. a material may be reused by a skinned and non-skinned mesh, meaning you need different shaders and so must collect and clone the materials (at least this is how it works in three.js).

I'm a bit worried about this... now if a top-level material is targeted we need to track down the nodes that use it, and possibly clones of the material created e.g. because of skinning. The three.js animation system generally targets nodes and their subproperties, and is based closely on Unity's so I'm guessing the situation is similar there.

Reusing materials is also important for an application, and having the authoring tool split identical materials in order to target them with different animations isn't ideal. On the other hand, if materials are reused it is possible to modify uniforms on a per-object basis while rendering. (This may be more complex for the client, but is achievable and likely to be worthwhile.)

I don't think easier implementation is a sufficient reason for targeting top-level objects with animation. It feels less natural conceptually, and the fact that it loses information about where materials could otherwise be reused worries me.

Top-level approach leaves the whole material targeted everywhere its used;

Potentially across multiple scenes, also?

[najadojo]

Extending Animations

Property animations can be added to an animation by adding the EXT_property_animation extension to any glTF animation. For example, the following defines an animation with two channels that modify the baseColorFactor and roughnessFactor of a mesh's material.

"animations" : [
    {
        "channels" : [ ... ],
        "extensions" : {
            "EXT_property_animation" : {
                "channels" : [
                    {
                        "sampler" : 0,
                        "target" : {
                            "node" : 3,
                            "path" : "mesh/primitives/0/material/pbrMetallicRoughness/roughnessFactor"
                        }
                    },
                    {
                        "sampler" : 1,
                        "target" : {
                            "node" : 3,
                            "path" : "mesh/primitives/0/material/pbrMetallicRoughness/baseColorFactor"
                        }
                    }
                ]
            }
        },
        "samplers" : [
            {
                "input" : 6,
                "interpolation" : "CUBICSPLINE",
                "output" : 7
            },
            {
                "input" : 8,
                "interpolation" : "LINEAR",
                "output" : 9
            }
        ]
    }
]

EXT_property_animation adds a channels list separate from the core specification's such that a different set of target values can be described.

target

The target object identifies the node or scene to animate using a node or scene property. Which property of the node or scene is identified by path.

path

In a typical engine at runtime glTF is transformed from a series of JSON object lists into a connected graph of engine objects. To represent the runtime instancing inherent in such a transform EXT_property_animation specifies animation binding at their runtime location.

For the following glTF:

{
    "materials" : [
        {
            "pbrMetallicRoughness" : {
                "baseColorFactor" : [
                    1.0,
                    0.0,
                    0.0,
                    0.2
                ],
                "roughnessFactor" : 0.0
            }
        }
    ],
    "meshes" : [
        {
            "primitives" : [
                {
                    "attributes" : {
                        "NORMAL" : 2,
                        "POSITION" : 1
                    },
                    "indices" : 0,
                    "material" : 0
                }
            ]
        }
    ],
    "nodes" : [
        {
            "mesh" : 0
        }
    ]
}

EXT_property_animation will interpret as if it were operating on the transformation:

{
    "mesh" : {
        "primitives" : [
            {
                "attributes" : {
                    "NORMAL" : 2,
                    "POSITION" : 1
                },
                "indices" : 0,
                "material" : {
                    "pbrMetallicRoughness" : {
                        "baseColorFactor" : [
                            1.0,
                            0.0,
                            0.0,
                            1.0
                        ],
                        "roughnessFactor" : 0.0
                    }
                }
            }
        ]
    }
}

Transformation follows each indexed object reference (i.e. "mesh": 0) and replaces it with the referred object and continues descending.

path is a JSON Pointer in the transformed JSON starting from the node or scene object and ending at a leaf property value. A path in the above document:

mesh/primitives/0/material/pbrMetallicRoughness/baseColorFactor

Both mesh and material are indexed object references that are followed from the original JSON. Circular references are acceptable in the transformed JSON but a path must always takes the shortest path from root to leaf.

This was a previous iteration description having node/scene based targeting. I think both approaches have their downsides; clarity of effect and difficulty of implementation in one place or another.

It'd be best to have a target description that is explicit where we can describe "everything with this material" or "just this instance of this material" but I've not encountered an animation system that is this descriptive. In three.js' animation system when targeting node.material.color if the material object is shared among multiple nodes the color changes on all even though that wasn't the target described.

The three.js proof of concept I did not attempt to collect up objects that would need to have clones based on an animation. Would the algorithm be simply if material is part of any animation assign a unique clone it unless it was previously cloned? It seems like it would be better to group them based on the key data and top level target (i.e. three blinking lights on my jet model, should be one animation on one material).

[scurest]

Perhaps it's worth advising what a producer should do for an animation that doesn't have any regular glTF channels since it's not immediately obvious. Current syntax requires a non-empty animation["channels"]. You can however use

"channels" : [
  {
    "sampler" : 0,
    "target" : {
      "path" : "scale"
    }
  }
]

because the spec says that channels without target["node"] properties should be ignored. (Both 0 and "scale" are arbitrary; scale is just the shortest path name).

edit: ah, I see this was discussed above.

[scurest]

The property need not be present in the original JSON; default or implicit properties can be animated.

Does this extend to animating KHR_texture_transform properties on a texture info object that doesn't already use KHR_texture_transform?

[najadojo]

Yes I think the language of the extension allows that. Though supplying a requires or uses extension should be necessary as those properties are undefined without the target extension.

[lexaknyazev]

@najadojo @bghgary @donmccurdy

Could you please elaborate a bit on the positioning and the life-cycle of this extension? I understand that this extension has EXT prefix so it can be defined and implemented as vendors wish. However it could become a de facto way of animating materials or lights much sooner than we could come up with a KHR extension or a new spec version.

I think that at some point, we'll want to have something very similar to this extension in glTF core. The current fallback design is clearly not optimal and imposes additional restrictions like existence of "node ": 0 and an unused sampler with two accessors. The question whether definitions of certain objects are animated or only their instances (like materials, cameras, or lights) should still be open even if it falls outside of the scope of this extension for now.

Moreover, I'm still concerned about (from comments above):

An existing implementation will fail to load if a new path is now part of the channels list.

and

It also seems node is not optional for existing engines.

since such behavior doesn't follow the schema.

[najadojo]

This extension was a proposal for something we were thinking about internally, unfortunately we are no longer perusing those plans. I started with EXT because I was able to prototype functionality in multiple engines and tools. There has been pretty good engagement in the discussion for this PR but I don’t believe enough to solidify the proposal. It may make the most sense to archive this PR until future additions to glTF core are underway.

[donmccurdy]

It also seems node is not optional for existing engines.

since such behavior doesn't follow the schema.

Added this as a suggestion for https://github.com/KhronosGroup/glTF-Asset-Generator/issues/414.

I think that at some point, we'll want to have something very similar to this extension in glTF core.

I think it would be useful to have a general syntax for extensions to use when adding animatable properties (the syntax in this PR seems reasonable, also see https://github.com/KhronosGroup/glTF/issues/1346) and if that requires changes to glTF core at some point we should consider it. But I do still have reservations about making things animateable-by-default, and would prefer that we consider allowing animation property-by-property as part of any new extension. For example, KHR_lights_punctual2 might add animation of punctual lights.

[lexaknyazev]

KHR_lights_punctual2 might add animation of punctual lights.

That's something I'd avoid, because it will likely lead to unmanageable fragmentation.

[donmccurdy]

@lexaknyazev could you explain further? In the long term, my opinion is that when we add any extension where animation is likely to be needed, that extension should define which of its properties can be animated, using syntax that is (a) part of glTF core, or (b) not in core but at least consistent across extensions. I only suggest KHR_lights_punctual2 here on the assumption that it's too late to add animation to KHR_lights_punctual.

Are you concerned about having extensions define animation, having inconsistent channel target syntax, or something else?

[donmccurdy]

/Bump. I think it would be helpful to have a general plan for (eventually) supporting additional animated properties. Example:

1. ???_property_animation extension is created, defining a syntax for targeting properties with animation, and listing properties in the existing spec and existing extensions for which implementations of the extension must support animation.
2. Future extensions should use the syntax defined by ???_property_animation to specify animation, if animation makes sense for that extension. For example, a hypothetical KHR_lights_area extension should declare which of its properties may be animated.

In retrospect, it would have been useful to have an animation syntax for arbitrary properties in place from the beginning, so that extensions like KHR_lights_punctual and KHR_texture_transform specs could include animation in a pre-defined way.

[donmccurdy]

A few more things I'd like to track here:

• Per #1518, using JSON pointers (or some other generic reference system) seem like a step in the right direction.
• [ ] Per #1520, consider allowing an animation channel to have multiple targets.
• [ ] Consider allowing animation of individual morph targets. E.g. meshes/{}/weights/{} or nodes/{}/weights/{}.

[pjcozzi]

@najadojo and authors - is this draft extension being used in any tools or engines? Does this need more work to get over the finish line? Or is this something that should be archived for future consideration?

Just tidying up open pull requests.

[najadojo]

No engines or tools that I'm aware of. The PR includes pointers to sample implementations in ThreeJS, Babylon, and Blender but all of those are out of date with their own masters. There has been a lot of discussion on the proposal here but we are no longer perusing this extension at Microsoft. Ideally someone else could take up the reins.

[pjcozzi]

@najadojo that for the prompt response and insight.

Potential archive this one, #1548, for someone to pick up when it is a priority for them.

[scurest]

It's implemented in ksons/gltf-blender-importer for material, texcoord transform, and node TRS properties.

[pjcozzi]

Thanks @scurest! Do you know if any engines plan to implement it / update their implementations?

[scurest]

No, for all I know I am the sole user.

[donmccurdy]

Per https://github.com/KhronosGroup/glTF/pull/1301#issuecomment-441108447 and https://github.com/KhronosGroup/glTF/pull/1301#issuecomment-451547256 I am still fully supportive of getting this extension, or something like it, completed. Recent requests from the community:

• How to Animate alpha in blender and export GLTF to three.js
• Can I read changing alpha value from the GLB file?
• Blender 2.8: is it possible to export animate color gradient to glTF?
• Export Material Node Animations in GLTF (or other) exporter

It would be helpful to know the opinions of other engines and authoring tools on this. Presumably actual support will depend heavily on the quantity of properties supporting animation (I would lean toward being conservative here?) but I will attempt a matrix below:

engine	feedback
threejs	will implement
Blender 2.8	will implement
BabylonJS	initial implementation here
BabylonJS Maya/Max exporters	?
Cesium	will implement
Adobe Animate CC	will implement
Unity	?
Unreal	?
...	?

Feedback welcome.

@MiiBond is animation of texture transforms something you can comment on? This seems relevant to Adobe Animate CC's glTF export, for example.

^~~I think we'll need a few more than two implementors engaged to make progress here. Otherwise we can set this aside for now and revisit in the future.~~ More implementors added above.

[MiiBond]

I'm waiting to hear back from the Animate CC guys. For Dimension, we don't currently have plans for this but I can see it becoming something we need eventually. I'm definitely in favour of an extension to provide this functionality. I am aiming to pass through all animation data, unaltered, from glTF->Dimension->glTF so, if this existed, I'd be at least implementing pass-through for this extension.

[pjcozzi]

CesiumJS will implement this, but we would likely not be in the first wave of implementers due to other priorities, but certainty 👍 from me to move this along.

[MiiBond]

I checked in with the Animate CC team and it looks like they will definitely need this. Animating texture transforms for sure and quite likely animating material params (e.g. opacity).

[donmccurdy]

Are the material params they need covered by the core spec? I'd expect that a discrete boolean 'visiblity' animation might be important in that use case, to avoid using opacity as a hack for "don't render this".

[MiiBond]

It's possible that we could go with visibility as that's cleaner but I'm not sure it simplifies things dramatically compared to checking for opacity == 0. Opacity does seem a bit hacky but, currently, Animate is using an object scale of 0 to make meshes disappear because it's the only property they can animate in glTF :)

[Jaswant99]

Hi I am from Adobe Animate Team. EXT_property_animation extension along with KHR_texture_transform will be quite useful to us. In Animate, we have an animation timeline where in each frame a shape can be added or removed. This is not the case in glTF where no new node can be added or removed in between the animation. To solve this currently, we are using a hack where we create node and mesh for each shape and use scale animation to hide and show such shapes when needed by making their scale 0. So we have a lot of meshes and nodes in the scene at one time which reduces the performance. To address this performance issue, we are planning to create a texture atlas and just change offset rather than creating new meshes. To do this we need KHR_texture_transform properties like offset, which can be animated using EXT_property_animation.

For a long term solution, we will need visibility animation or something like start time and end time for nodes i.e. to add the node at time >= start time and remove it on time >= end time.

[donmccurdy]

Thanks @Jaswant99! Animating KHR_texture_transform should definitely be within scope for this extension. I'm unsure whether visibility (independent of opacity) is or not.. I don't think the glTF spec should have a visibility property on nodes – is it OK for this extension to animate something that isn't a JSON property? 😕

Copying a couple other open questions from https://github.com/KhronosGroup/glTF/pull/1301#issuecomment-451547256 –

• should animation channels be able to have multiple targets? (unsure)
• should morph targets be able to be animated independently? (leaning yes)

[Jaswant99]

I don't think the glTF spec should have a visibility property on nodes – is it OK for this extension to animate something that isn't a JSON property?

so are you implying that we will need a node extension for visibility just like khr_texture_transform was added for material?

[emackey]

To turn off visibility, it may be worth calling out that scale is allowed to become 0, 0, 0, and implementations are welcome to optimize this by turning off rendering of the entire child-nodes tree (as there's no recovering visibility from scale zero by any child nodes).

Implementations that don't support this optimization will still do the right thing when all the geometry becomes degenerate. This is better than for example rasterizing a ton of fragments that all fail an alpha test, as opacity animation would do.