Smithay
RFC: Replace user data maps with ECS
This is proposed for 0.5. Please just release Smithay 0.4 and then we can delve into implementing this.
The current problem
At the moment Smithay relies on interior mutability for all data stored in the user data of protocol object wrappers. This makes the data storage effectively static. If a user of smithay wanted to attach extra information to an object it would be impossible on the object itself and storage elsewhere would be needed. This is effectively required for extension protocols.
For cases where attaching extra data to an object is desired, Smithay tries to solve this with the UserData and UserDataMap types to allow extra data to be bundled. There is a catch to this, it is often confusing what type needs to be used when fetching the data (since the type is effectively the key). Most parts of the wayland module which do this either expose newtype wrappers for data or a more raw form of the data such as a Mutex<XdgToplevelSurfaceRoleAttributes> (which is typedefed as XdgToplevelSurfaceData. Like mentioned above, extension protocols can use these data maps, all the shell modules in smithay use the user data to store internal information.
It is also extremely easy to cause Arc cycles with interior mutability. Arc::new_cyclic can help with that issue but it makes some patterns impossible.
Interior mutability also means the codebase is littered with Mutex, RwLock and Atomic types in user data. Atomics are relatively cheap on modern hardware. Acquiring a guard on a Mutex or RwLock is an additional cost that has no real benefit considering the wayland server is typically run on a single thread. Plus a display can only be dispatched on one thread at a time (Display::dispatch takes &mut T) so unless work is delegated to worker threads most of the time we are locking and unlocking wastefully.
Why ECS?
ECS has several tools which make solving the problems easier. The main one being that attached data would be centralized in the compositor state. The compositor state is already dispatched to every corner of the wayland protocol handling. Therefore it would only be logical to fully utilize the compositor state.
ECS allows the data attached to an entity to be dynamic. This means a user of smithay can attach any data that is wanted to the objects. Libraries that extend smithay can also use that data storage. A compositor developer, smithay and a library can all share the same data storage.
Wayland's protocol also lends itself well the ECS. Extension protocols are quite common and can the data for each extension be represented as a component attached to an object.
For a compositor implementation it may also be desirable to have a group of objects be considered the same entity. This means a toplevel surface, viewport and wl_surface would all refer to the same entity id.
Testing the idea
https://github.com/i509VCB/smithay-ec
I wrote a small (not functional, but wrote out enough to test it) compositor library similar to Smithay using ECS. I used hecs in this case which seems to be flexible enough for what Smithay is trying to do. This is to show that wayland-rs would not require any significant changes and most of the work would occur in Smithay.
Queries in hecs allow the compositor and implementation to query several components at once. This can resolve some of the borrow checker issues (although hecs does pay with some unsafe internally for that convenience).
In particular ECS allowed me to make the pre commit, post commit and destroy callbacks in the compositor module able to reference the compositor state. This is the "system" part of ECS. Other wayland protocol handlers would probably grow system notification mechanisms (such as seats and outputs).
For Smithay it can use internal components and then expose publicly accessible components (by making the component public) for users of the library.
Possible concerns
There are of course some caveats. Multi-threaded worker threads are a little more complicated if compositor state needs to be accessed, but arguably you might race the compositor state if you query data like this. hecs does allow running a query on a world with runtime borrow checking and spawning entities off thread if this was desired.
This could also make Smithay's wayland frontend more opinionated. I doubt this will be a huge issue though since nothing stops data queried from ECS to be stored and managed outside of ECS by the compositor.
Hierarchies are not one of ECS's strong suits, but it can be done with entity ids. For subsurfaces the implementation would look very similar still.
Possible design requirements
It may be desirable to get the WlSurface of an XdgToplevel and vice versa. It should be easy to query that from the entity.
Protocol object wrappers would need to track an internal Entity id. The entity id is an immutable generational id so it would be cheap to clone and require no interior mutability. This however means the implementation or user would need to unwrap the entity id or the storage would allow using a protocol object as a key (but it would internally unwrap).
