glTF
glTF copied to clipboard
KhronosGroup
Create a specification for rigid body physics
A number of people and companies have expressed an interest in having a GLTF extension which can describe rigid body physics.
A number of popular physics engines are widely used, each with a differing API and data serialization format. While many concepts between them are the same, there also tends to be significant differences between them as well.
Past efforts to create a standardized physics data definition have ended up with something so watered down that it was of questionable value, or so highly specific to one physics engine, that it could not be easily used by others.
I see the goal of this effort to create a data spec which represents a 'union' of the features of a number of popular physics engines; such that those things which are in common between them use a shared DOM (data object model) and those properties which are unique to each can still be specified and, conceivably, ignored by other physics engines which do not need them.
Even just the concept of a 'physics engine' can be challenging, as the definition can be wide ranging in scope. Some physics engines support soft bodies, particle systems, fluid dynamics, complex machines, and more.
While it would be desirable to support all features for all physics engines, that is probably unrealistic as an initial goal.
I would like to set the phase one goal to simply defining the basic components shared by any rigid body dynamics system; which is largely collision geometries, physical material properties, simulation settings, rigid body settings, collision filtering, and joints.
We should probably identify features that go beyond this, but leave them for future extensions efforts. Future work should also address the specific needs of robots which include cameras, sensors, grippers, transmissions, and a lot complex machine data.
Typically a DOM for a physics engine is similar to the scene graph you would see for rendering data; there are meshes and geometries which can be instanced at different locations and scale and they can be linked together with joints. There may need to be a way to specify a named binding between a physics representation and the visual representation it is associated with. Likely a parallel scene graph which simply describes those bindings will be needed.
The basic components of any rigid body system can be described in the following outline:
- Simulation properties
- Each physics engine defines certain global properties which control how that system performs. Many of these properties tend to be unique for each different physics engine supported. Includes things like gravity, solver iteration count, and other global settings.
- Data Resources (objects with large amounts of binary data)
- Triangle meshes
- Convex Hulls
- Heightfields
- Physics Materials
- Defines physical properties of a material surface, friction, etc.
- Geometries (physical geometric representations for rigid bodies or parts of a compound rigid body)
- Plane
- Box
- Sphere
- Capsule
- Cylinder
- Triangle Mesh
- Convex Hull
- Heightfield
- Rigid Body (can be static, dynamic, or kinematic)
- Each rigid body has one, or more, geometries and their associated unique properties; such as what material to use, what scale, etc.
- A large array of physical properties are associated with each rigid body (mass, inertia tensor etc.)
- Aggregates
- A collection of rigid bodies which can be treated as a single group.
- Joints (a joint is a type of constraint which joins two rigid bodies together)
- Fixed
- Spherical
- Revolute
- Prismatic
- Distance
- D6
- Body Pair Filters
- Defines a sequence of object pairs which should or should not interact in the physics simulation.
- Collision filter logic
- Describes the code/logic for how collision filtering between two rigid bodies should be interpreted.
The initial phase of this project will simply be requirements gathering. We have some graduated students who will be evaluating a number of well known physics engines to understand the data object model requirements for each. Once those are gathered we can try to create a 'union' of these specifications as well as identify engine specific properties. The last phase will be to actually create the schema itself and submit it for review. This process could take weeks to months to complete, and is only starting today.
Physics engines under consideration are:
- PhysX SDK : from NVIDIA
- FLEX SDK : from NVIDIA
- Bullet SDK : (open source by Erwin Coumans)
- Newton physics engine : (open source by Julio Jerez)
- Havok : from Microsoft
- Mujoco : Commercial physics engine, popular within the robotics community
A rigid body physics extension has already been started as #1104. Maybe we can build from there?
@jratcliff63367 thanks for kicking off this effort, it is much appreciated! Adding @akhabbaz and @rms13 to the conversation.
@Moguri it would be great to see if there are collaboration opportunities and perhaps end up with one extension.
Yeah, I'm fine with merging the efforts. That one seemed specific to Blender only? I think the larger goal is to be more general to a wide range of physics engines.
@jratcliff63367 That extension is not meant to be Blender specific, and tries to generalize physics settings when possible. The reason it has the BLENDER prefix is because blendergltf is currently the only exporter using it. When multiple exporters and importers start supporting it, it can be promoted to an EXT or KHR extension.
Ok, this seems great then. We are looking for engineering resources to continue this effort so that it can encompass a number of popular physics engines as well. I'm trying to rustle up some college interns now.
Here's an example of a skeletal mesh and rigid body system I would like to be able to fully describe in GLTF.
https://www.youtube.com/watch?v=hw1TpxfglrE&feature=youtu.be
I see the goal of this effort to create a data spec which represents a 'union' of the features of a number of popular physics engines; such that those things which are in common between them use a shared DOM (data object model) and those properties which are unique to each can still be specified and, conceivably, ignored by other physics engines which do not need them.
While it would be desirable to support all features for all physics engines, that is probably unrealistic as an initial goal.
I am a bit nervous about this broad extension scope. glTF has so far had a lot of success with focusing on common features that can be implemented consistently across many engines. Is there a (still useful) approach to physics that would allow us to do the same?
If a particular physics engine wants to jump in and propose their own extension (building on this one), e.g. BULLET_physics_extras, they can certainly do that. But if everything is defined in a single superset physics extension, it becomes very hard to ensure consistent implementations, as glTF's extensionsRequired and extensionsOptional are no longer useful indicators of support.
I was envisioning something where you specify the common properties and anything engine specific can be added on as additional key/value pair tags. I'm not really familiar with JSON schema specs yet, but that's fairly common in XML specs.
Sometimes the motivation to incorporate physics in a graphics specification comes from an observation that it's nice to represent arbitrary polytopes with vertices edges and faces, and hey, graphics representations do that too, but the end goals are very different, and I think that ultimately the equivalence is superficial. I say this because I did a couple of physics engines at LucasArts where initially I went in with the goal of sharing graphics data with the physics engine, but it never turned out that the data could be shared in a practical sense. Physics representations by necessity are simpler and tend to bound an object rather loosely, and hugely benefit from implicit formulations. I know I'm belaboring the obvious, but I wanted to be clear about where my objection is coming from.
That said, I'm not meaning to rain on the parade. To the contrary, I am enthusiastically in support of the creation of a physics schema that can be independent of a particular engine. I've always been hopeful and then disappointed when things like PAL appear and then don't really gain traction. I just don't feel that gltf shouldn't take on more than representations of graphics state and things to render. I'm wondering if it might be a good idea to form a Physics Interchange oriented group to come up with a specification, independent of gltf, for broader use.
Long story short, all the goals you've laid out are awesome, I think they deserve their own, independent of gltf, effort.
Please, take BepuPhysics for consideration to the list of physics engines. It's one of the few 100% C# free physics engines around
@jratcliff63367 What's the current state of this proposal? I would like to move #1104 along so we at least have rigid body physics, but I would still prefer to collaborate on physics in general.
Maybe it makes sense to break this general physics extension into multiple physics extensions (e.g., rigid bodies, soft bodies, constraints)? This may make it easier on consumers if physics isn't an all or nothing deal (e.g., start with rigid bodies, then start implementing something like constraints). A consumer could ignore most of the extension, but it is odd to list support for an extension if support for large chunks are missing.
Here is the current status. I'm sorry it isn't further along, especially with the JSON stuff, but I'm also using this as part of a work project and that project has some different constraints and priorities. In particular, for the work stuff I have been told I need to support google protobufs, so there's been more progress on that than with the JSON schema at the moment.
That said, progress on the overall data-object-model has moved pretty far. I have a data definition for most of basic rigid body types. The missing parts are that I have not yet fleshed out all of the properties for the various joint types. That is still a todo item. The data definition includes a full hierarchical scene graph, easy to parse and understand.
Regarding properties specific to individual physics engines, I have added a generic 'property bag' called 'AdditionalProperties' where you can add key/value pairs for individual physics engines.
For example, say 'Bullet' has a bunch of properties specific to it when defining a scene. You can just add those properties to the 'AdditionalProperties' array in the Scene class. Same thing for 'PhysX' or any other physics engine.
Here is a link to the current data-object-model in a Google spreadsheet. The major missing component is detailing all of the properties for each individual joint type.
https://docs.google.com/spreadsheets/d/118I5kdu2XT-6wfCG044937xfEKDyX2oNg04G8Wqi6o0/edit?usp=sharing
I am not addressing soft-bodies with this spec. My current goal is to limit this specifically to rigid bodies.
I have a tool called 'CreateDOM' which parses this google spreadsheet and, the goal, is that it will automatically generate all of the C++ binding code and serialization schemas necessary to operate on the data.
For example, here is the auto-generated C++ header file for the spreadsheet linked above:
https://github.com/jratcliff63367/CreateDOM/blob/master/PhysicsDOM.h
It is important to note that this C++ header file is in POD format (plain-old-data). It has no depedencies on the STL; it's just the raw data and is safe to pass across DLL boundaries.
The implementation class (which is still a work in progress), is used to build, copy, move, iterate, manipulate, insert, delete, and otherwise modify the data.
Here is a link to what the implementation/reflection class looks like. This is not yet complete, and is still undergoing a lot of improvements. This code, too, is 100% auto-generated from the Google spreadsheet specification.
https://github.com/jratcliff63367/CreateDOM/blob/master/PhysicsDOMImpl.h
I realize that the focus for GLTF is JSON but, as I said, my internal priority is to get Google protobufs support.
Here is the auto-generated protobufs schema for the Google spreadsheet above. It is the design goal that this tool will, likewise, auto-generate the GLTF JSON schema as well.
https://github.com/jratcliff63367/CreateDOM/blob/master/PhysicsDOM.proto
My next tasks are:
- Finish adding all of the properties for the various joint types
- Finish auto-generating the implementation class; which lets you construct, move, copy, insert, delete, and iterate on the contents of a DOM representation.
- Auto-generate the GLTF compatible JSON schema
- Auto-generate the JSON serialization and deserialization code
- Finish implementing the demo app which can load any fully specified DOM and simulate it in a physics engine (some progress has already been made on this task)
I have some other requirements as well for protobufs and XML support, which are not important to this group but necessary for my internal project. I also intend to support converters for popular robotics file formats such as URDF, SDF, and Mujoco.
@jratcliff63367 It really seems like you're making your own physics standard here with the plan to eventually write a glTF extension to handle that standard. While I would still prefer to collaborate, I also prefer the smaller, more focused scope of the currently proposed BLENDER_physics extension. Such an extension is easier to support (both by exporters and importers), which helps improve adoption. BLENDER_physics is also usable now (including one known exporter and one known consumer) while this extension still appears to have a fair amount of work left.
As such, I plan to continue to develop the BLENDER_physics extension. Perhaps your extension will eventually come to replace it as a more complete physics representation. The two extensions may also come to exist side-by-side as a simple option and a complete option. Hopefully there will still be opportunity to collaborate on physics in the future.
I was trying to create a completely generic representation of rigid body physics. If what I have in that spreadsheet doesn't meet that goal, then I have failed. All concepts in there are very generic and should work with most major physics engines; at least the ones I have used which are PhysX, Bullet, Newton, and Havok.
I am not trying to 'create my own'. My goal was to create a generic rigid body DOM which could be used by GLTF. If I have somehow failed in that, I'm not sure where. That spreadsheet is quite basic and simple at this point.
@jratcliff63367 Maybe I didn't explain myself accurately, but "create a generic rigid body DOM which could be used by GLTF" is what I was getting at when I said you were creating your own standard. This could be it's own thing (i.e., a stand-alone physics description format), and it feels like it already is with XML, protobuf, and JSON implementations. I feel glTF is just acting as a wrapper to this format. I find the DOM you have so far to be overly generic and too inclusive for a lot of cases. A complete spec like this is great for some use cases and useful to have (as @meshula mentioned), but I believe a simpler spec also has its uses (easier to implement, still covers a lot of cases).
As a separate conversation, I noticed your spec does not have a way to lock motion on particular axes. Bullet calls this linear factor and angular factor, and may be useful to include.
PhysX supports locking motion as well, but I didn't think that was a generic concept. Every object in the spec is a 'Node' and every node can have a set of 'additional properties'. The idea is that engine specific concepts can be expressed under the additional properties fields. That said, if locking motion on an axis is a generic enough concept, it could be added to the base spec.
To clarify what my task at hand is. For an internal project at work I need a way to express rigid body physics in a generic way that can be run on multiple physics engine backends.
I believed this was also a goal of the GLTF project, so I felt there was an opportunity to 'kill two birds with one stone', since the goals seemed largely the same.
I do not think my current spec is complicated, in fact I thought it was quite simple and generic, but capable of expressed many complex rigid body systems.
If there is no desire for this effort to be integrated into GLTF, I can stop contributing to this effort here. My tasks for my work project remain the same.
I believed this was also a goal of the GLTF project, so I felt there was an opportunity to 'kill two birds with one stone', since the goals seemed largely the same ... I do not think my current spec is complicated, in fact I thought it was quite simple and generic, but capable of expressed many complex rigid body systems.
In my opinion @jratcliff63367's proposal is both simple and generic for the expressed goal of providing a superset of physics engines' capabilities. In contrast, glTF has so far taken a "subset" approach — as one example, subject to some debate, we've limited assets to 4 influences per vertex for skinning, because some key engines cannot support more. This has the advantage that glTF assets can be expected to work very broadly, but is clearly less expressive.
Whether that same "subset" approach is right for physics, I don't know. There are features described here that cannot be expressed in JS physics engines like Cannon.js or Oimo.js, but JS frankly lags behind on good physics implementations, and I don't really expect that to change until our browsers' GPU APIs improve.
In my opinion that is the crux of the difference between the BLENDER_ proposal and this one. Having both may be a reasonable compromise. Or perhaps there is some way for one to "extend" the other? I'm not quite sure how to handle the protobuf and XML requirements, and would defer to you on that.
The alternative is that some things may be expressed as higher-abstraction wrappers around glTF, as 3D Tiles do to combine geospatial metadata and glTF assets, rather than as extensions.
@jratcliff63367 Just to be clear, I am not saying that there is not a place for this extension. I am just one voice, and I find a more limited physics subset to work better for my current needs. I am sure other people would love a more complete physics spec. Also, I would not say that your spec is complicated; it just includes more (e.g., joints) than engines may want to (or even can) implement. I would say "complete" or "comprehensive" is a better description than "complicated."
I would like to give my humble opinion about rigid body support on glTF;
I agree with @meshula in that maybe trying to add any kind of rigid body representation to glTF might go beyond what's really needed. Sometimes we must remember the KISS philosophy. Good things are tiny.
Let's think in terms of Model-View-Controller; Everybody would consider having "Employee" profile information embedded within the photography of that employee as a crazy design.
That being said. Having rigid body information embedded in the glTF file makes more sense from the point of view of how current artistic pipelines work today: I can see a 3D artist creating a 3D model, and including a rigid body layer so the artist can export everything in a single shot.
Another matter is whether to include the rigid body information within the gltf schema or not.... indeed, I would preffer to have everything in a separated schema AND a separated file.
Consider a physics engine having to load a single RigidBody+Visuals file, it needs to deal, and skip all the visual/graphic elements. The same happens for the graphics engine, it needs to skip the rigid body information data.
With separated files, each domain would load exactly what it needs, without interferences from the other.
@jratcliff63367 @Moguri since I do not have the physics domain knowledge, I am relying on you two to guide this.
Based on @donmccurdy's comment, it sounds like it could be fine to have a very simple Blender physics extension and the full-featured extension - if the former isn't easily designed as a subset of the later, or the former can be done much more quickly. Does this make sense or not?
It's not a perfect analogue, but perhaps this is like glTF common materials vs. PBR. Both are valuable and expected to be widely implemented.
@jratcliff63367 @Moguri we discussed this in the Khronos 3D Formats working group. The group is OK with having two extension, especially this early in the glTF physics ecosystem where we need to get more implementation experience to flush things out - it could benefit from different approaches.
Let me know if there is anything I or the group can to do facilitate things moving forward.
The appeal of this to me is my users could set up collision shapes in a modeling program and export them in GLTF, giving them finer control than the auto-generated shape options we support in our editor.
This extension should be focused purely on GEOMETRY, for the artist's purposes. Joints, physical properties, and collision behavior definitions should definitely be kept out. If this is to be useful at all I think it needs to just define a physical shape (procedural or polygon / hull) for the whole object. Otherwise it turns into a physics engine serialization system.
In fact, the reason there is no standard physics shape format is likely because people just can't help themselves from expanding it into joints, physical properties, etc.
We use these shapes:
- box
- sphere
- cylinder
- capsule
- convex hull (standard GLTF mesh with positions only)
- polygon mesh (standard GLTF mesh with positions and indices only)
- cone
- chamfer cylinder
- compound (aggregate of any of the others)
If there are other shapes in other physics engines please list them.
I would leave heightfields out, because there are too many options for how those can be stored and used, and terrain is likely to be an engine-specific feature anyway.
For each shape, you need an orientation defined, so the standard mat4 or quat / position works fine. Is scale allowed?
Are we defining a collision shape for the entire GLTF scene, or do you want to define shapes on a per-node basis?
Do convex hulls need indice info or is a cloud of vertices enough?
Here's a sample of what it might look like:
collisions {
{
"shape:" "BOX",
"translation": [4,6,2]
}
{
"shape:" "CONE",
"rotation": [-0.70710678118654746,-0,-0,0.70710678118654757],
"scale": [2,2,2],
"translation": [1,0,0]
}
{
"shape:" "CONVEXHULL",
"scale": [1,1,1],
"translation": [1,0,0],
mesh: 3
}
{
"shape:" "COMPOUND",
"collisions": [0,1,2]
}
}
Since GLTF is a scene file format, even though I use it exclusively as a model format, it makes sense that different objects in a scene would have different collision shapes. Therefore I suggest adding a "collisions" or "shapes" tag to the node structure that indicates one or more physics shapes:
nodes {
{
...
collisions: [0,3,6]
}
}
This design would eliminate the need for compound / aggregate shapes, which would get unnecessarily complicated if they were nested.
I'm coming from Newton, but the features of Blender largely line up: https://docs.blender.org/manual/en/latest/physics/rigid_body/properties.html
- Box
- Sphere
- Capsule
- Cylinder
- Cone
- Convex hull
- Mesh
Again, I would just focus on the collision geometry, because physical properties like mass can easily be typed in an engine's editor, and the reusability there is pretty much non-existent.
Just focusing on the above, one question that needs to be addressed is whether a sphere can have a non-uniform scale, or does it just accept a single radius value? Newton supports this but reading the above, it looks like Blender does not. http://www.newtondynamics.com/wiki/index.php/NewtonCreateSphere
Same question applies to cylinders, cones, and capsules (on two axes). Should it be something like this?:
{
"shape:" "CONE",
"height": 2.0,
"radius": 1.0
}
Perhaps the scale should be stored the same way as in nodes (scale property or in matrix) but it should be specified that only the first component will be used in case of a sphere, and only the XY components will be used for a cylinder's diameter and height (if non-uniform scales are not to be supported). That takes care of it, while allowing for future expansion or wiggle room for those who choose to veer outside the spec:
{
"shape:" "CONE",
"scale": [2.0, 2.0]
}
What is the appropriate way to store a sphere?
{
"shape:" "SPHERE",
"radius": 0.5
}
{
"shape:" "SPHERE",
"scale": [0.5]
}
{
"shape:" "SPHERE",
"scale": 0.5
}
I lean towards making a firm rule that non-uniform radii are never allowed. GLTF seems to be at its best when its limitations are embraced, and I don't think I have ever encountered a situation where it was actually needed in all my years of engine development.
@JoshKlint I plan to revive #1104 as a collisions-only extension and strip out rigid body properties (e.g., mass) for another, separate extension. If you're willing to collaborate, I can prioritize getting a draft of that extension published again.
I am willing to help in any way. I don't have any experience writing extension specs but I am interested in putting some effort into it.
Docs from 3ds max, looks like it aligns pretty well: https://knowledge.autodesk.com/support/3ds-max/learn-explore/caas/CloudHelp/cloudhelp/2019/ENU/3DSMax-Simulation-Effects/files/GUID-B0C87F87-6E4B-41C4-B0EB-94ACE223E9C9-htm.html
Drilling down into this, an array of positions isn't really a complete description of a convex hull at all. Blender has a minimum triangle area value, and Newton has a tolerance value used for vertex merging: http://newtondynamics.com/wiki/index.php/NewtonCreateConvexHull
The point is, a list of vertices is not sufficient to create a consistent reproducible convex hull. A real definition of a convex hull would consist of a vertex array and an array of faces / polygons with a variable number of sides. So should the hull construction step take place before the hull is saved in the file, or after the raw data is loaded from the file?
Each face could be defined as a plane equation, or as a list of indices into the vertex array. If a list of indices are provided, the plane equation would be redundant because it can be calculated from the first three vertex positions. That might mean that this extension is dependent on completion of the n-gon stuff others are defining (https://github.com/KhronosGroup/glTF/pull/1620). Is a GLTF mesh really appropriate to try to cram into a convex hull definition at all?
I'm afraid if a full definition is not provided then a lot of extra values will be tagged in like "Blender minimum triangle area" or "Newton tolerance" leading to inconsistent results.
I'm sure many implementations, including our own, will initially ignore the indice list and reconstruct the hull from the vertex list, but I think things should move towards an unambiguous definition of a convex hull.