thirdroom icon indicating copy to clipboard operation
thirdroom copied to clipboard
verified publisher icon matrix-org

Manifold Editor

Open robertlong opened this issue 3 years ago • 4 comments

The Manifold Editor is our game engine environment / asset editor built around the glTF standard. Both the interchange and authoring format will utilize glTF and should be capable of producing content for Third Room as well as exporting glTF content for use elsewhere.

The todo list below represents the tasks needed for a feature-complete 1.0 release. We plan on tackling this list in chunks, focusing on iteratively improving our demo environments.

  • [ ] Scene Hierarchy Panel
    • [ ] UI
      • [x] Hierarchy Panel
      • [x] Tree View
      • [ ] Context Menu
        • [ ] Rename Entity
        • [ ] Delete Selection
        • [ ] Duplicate selection
        • [ ] Copy Selection
        • [ ] Paste Selection
        • [ ] Group selection
      • [ ] Create Entity Button / Dropdown / Combobox
        • [ ] Pick from list of prefabs
    • [ ] State (Game Thread -> Main Thread)
      • [x] Scene hierarchy
      • [x] Selected / active entities
      • [x] Entity names
      • [ ] Prefabs
    • [ ] Commands (Main Thread -> Game Thread)
      • [x] Set Selection
      • [x] Add Selection
      • [x] Toggle Selection
      • [ ] Focus Entity
      • [ ] Rename Entity
      • [ ] Reparent Entities
      • [ ] Remove Entities
      • [ ] Add Entity
      • [ ] Duplicate entities
    • [ ] Misc
      • [ ] Figure out cross-thread Tree View canDrop implementation
  • [ ] Properties Panel
    • [ ] UI
      • [ ] Properties Panel
      • [ ] Input Field / Row
      • [ ] Input Section / Group / Component
      • [ ] String Input (string)
      • [ ] Checkbox Input (boolean)
      • [ ] Select Input (enum)
      • [ ] Number Input (number)
      • [ ] Vector2 Input (vec2
      • [ ] Vector3 Input (vec3)
      • [ ] Euler Input (euler/quat)
      • [ ] Slider Input (number)
      • [ ] Color Picker Input (color)
      • [ ] File Input w/ DnD support (file)
      • [ ] Property Panel Button (command)
      • [ ] Add Component Button / Dropdown / Combobox
      • [ ] Remove Component
    • [ ] State (Game Thread -> Main Thread)
      • [ ] Selected / active entities
      • [ ] Active entity components and initial values
      • [ ] Component schemas
    • [ ] Commands (Main Thread -> Game Thread)
      • [ ] Add Component
      • [ ] Remove Component
      • [ ] Set component property
      • [ ] Set Transform properties (relative to active entity)
    • [ ] Components
      • [ ] Transform
      • [ ] Renderable? (More likely components for each renderable type which update renderable when edited)
        • [ ] Lights
        • [ ] Cameras
        • [ ] Meshes
      • [ ] Rigid Body
      • [ ] Audio Emitter
      • [ ] Spawn Point
  • [ ] Assets Panel
    • [ ] UI
    • [ ] Prefabs
      • [ ] Entity
      • [ ] Cameras
        • [ ] Perspective Camera
        • [ ] Orthographic Camera
      • [ ] Lights
        • [ ] Directional Light
        • [ ] Point Light
        • [ ] Spot Light
      • [ ] Audio
        • [ ] Audio Emitter
      • [ ] Meshes
        • [ ] Box
        • [ ] Cylinder
        • [ ] Cone
        • [ ] Capsule
        • [ ] Plane
        • [ ] Circle
        • [ ] Dodecahedron
        • [ ] Icosahedron
        • [ ] Octahedron
        • [ ] Ring
        • [ ] Sphere
        • [ ] Tetrahedron
        • [ ] Torus
        • [ ] TorusKnot
      • [ ] Misc
        • [ ] Spawn Point
    • [ ] Assets
      • [ ] Add to room state on upload?
      • [ ] gltf
      • [ ] images
      • [ ] audio
  • [ ] Viewport
    • [ ] Fly controls
    • [ ] Mouse selection
    • [ ] Transform controls
    • [ ] Grab controls
    • [ ] Helpers
    • [ ] Command Keybindings
      • [ ] Copy / Paste
      • [ ] Delete
      • [ ] Grab
      • [ ] Translate, rotate, scale
      • [ ] Transform space
      • [ ] Transform origin
      • [ ] Duplicate selection
      • [ ] Focus selection
      • [ ] Add/remove from selection
  • [ ] Toolbar
    • [ ] Transform controls mode
    • [ ] Transform space
    • [ ] Transform origin
    • [ ] Snap settings
    • [ ] Grid settings
    • [ ] Save
    • [ ] Publish
    • [ ] Export
  • [ ] Scene Editor (Compose a scene using referenced glTFs, save root gltf with references to assets)
    • [ ] Set opened glTF as scene
    • [ ] Render glTF hierarchy but disable transform / reparenting of referenced glTFs
      • [ ] Use glTF MX_ref/id extension for adding components to nodes
    • [ ] Save root gltf
    • [ ] Export entire scene as glb
  • [ ] Asset Editor (Edit glTFs and export modified glTF)
    • [ ] Set opened glTF as active scene
    • [ ] Render entire glTF hierarchy
    • [ ] Allow for glTF modifications
      • [ ] Use glTF MX_ref/id extension for adding components to nodes
      • [ ] Use glTF MX_ref/id extension to reference individual nodes when reparenting / deleting etc.
    • [ ] Save root gltf with modifications / references
    • [ ] Export entire scene as glb
  • [ ] Serialization
    • [ ] ECS -> glTF
    • [ ] Reference external glTF files
  • [ ] Deserialization
    • [ ] glTF -> ECS
    • [ ] Load referenced glTF files
  • [ ] Import
    • [ ] Load gltf
    • [ ] Generate thumbnail
    • [ ] Save to room state
  • [ ] Export
    • [ ] Export entire scene as single glb

robertlong avatar May 17 '22 18:05 robertlong

Hierarchy Panel and Properties Panel Design:

Editor state is shared from GameWorker to MainThread in two SABs

  • First is the same TripleBuffer as the render thread (hopefully at least, need to figure out sync mechanics since we swap buffers there, may need to be another triple buffer with just hierarchy data)
    • This would contain: parent, firstChild, prevSibling, nextSibling and a new hierarchyChanged flag used in the UI to perform efficient updates to the tree view component
  • Second is the active entity buffer
    • This SAB is included in the selection changed postMessage any time the active entity changes
    • That post message already contains the selected entity ids and their initial values. We'd remove the initial values though and replace with this SAB
    • This SAB would contain:
      • The values for every editor exposed component on the active entity
      • Including a flag for any property that changed which forces the component to rerender. This flag is set by checking each property against the prev SAB value and the current value

The first SAB is used to render the hierarchy panel and the second is for the properties panel.

Strings and other non-serializable fields are sent via post message as property changed events.

Every game tick we iterate over all the editor exposed properties of the active component and check for changed properties and copy to the SAB. If the field is a complex value (string, etc) we check for equality and then send a property updated post message. Dirty flags are reset at the beginning of each tick. If the active entity changes we create a new SAB and post message to the main thread. Getters for component properties are defined in component schemas. Complex equality operations should be defined here as well.

For the hierarchy data we set that dirty flag inside our transform component methods on the game thread. All of this data is stored in a SAB and syncs like any other triple buffer. No post message required.

For node names we'll want to create a component for storing names of nodes that will also store a dirty flag. We'll post message any changes to names from the game thread to main thread.

The main thread hierarchy view iterates over all the hierarchy info every animation frame and mutates its representation of the scene graph. It can be efficient about this by only updating the parts that actually changed.

The main thread property panel uses the component schemas it gets from another post message based API and the active entity buffer to render the component properties. Because it reads from a SAB, the inputs will be super responsive for any component property changes. Even if we're in play mode and observing an entity while it's in motion or animating, etc.

Big design points here are:

  • Less postMessage use == less GC pressure
  • Animation frame synchronized SABs == highly responsive UI even when lots of things are changing

All mutations from the Editor UI are made via postMessage to the GameThread. Setters for these properties are defined in component schemas. Creating entities will use prefabs. Prefabs are just functions that create entities with pre-assigned components. We'll display a list of prefabs to create when clicking on create new entity. Creating a new entity is accomplished via a postMessage with the prefab id and any initial properties sent from the main thread to game thread.

robertlong avatar May 17 '22 22:05 robertlong

Hi I had a question regarding the direction of how the Editor will be integrated.

Mainly I want to know if there has been any thoughts around being able to use the editor while in XR. It would be great to be able to directly create UGC and manipulate the scene without ever having to exit VR for example. This is especially useful in a collaborative setting where mutliple people can work on world or experience on any device they're using.

Some great inspiration for this would be for example NeosVR, where almost everything beside assets are created in-game. here's quick video showcasing how the scene editor for example works: https://youtu.be/dqsjidSmlJk?t=197

hazre avatar Jan 22 '23 01:01 hazre

Hey @hazre we're slowly working towards a collaborative XR editor but there are some smaller milestones that we plan to reach beforehand. Our next step is to finish the property panel which will allow you to start inspecting all of the data in your scene by pressing the tilde key. Then we'll add single player editing. Then multiplayer and so on. Editing in XR is on our roadmap though!

robertlong avatar Jan 24 '23 16:01 robertlong

Hey @hazre we're slowly working towards a collaborative XR editor but there are some smaller milestones that we plan to reach beforehand. Our next step is to finish the property panel which will allow you to start inspecting all of the data in your scene by pressing the tilde key. Then we'll add single player editing. Then multiplayer and so on. Editing in XR is on our roadmap though!

Oh okay sounds good, I just didn't see it on the issue itself, so I thought it wasn't considered. Glad to hear that's on the roadmap. I do wonder though how that's going to work if the desktop UI is in React; would the XR UI be in WebGL? if so, I feel like maintaining two different UI systems might be troublesome, maybe there should be some thoughts in place on a unified UI that works on all platforms?

hazre avatar Jan 24 '23 16:01 hazre