Entity Keys

[pablomayobre]

A common requested feature is a way to store Entities (or references to Entities) inside Components while keeping the data serializable.

That is, the next startup the Component data can be deserialized and the references to the Entities can be maintained without added complexity or data duplication.

We have decided to use a running number using integer values (starting from the smallest possible integer and increasing by one). But leave the option of changing the key generator and a way to preserve the state when deserializing.

[pablomayobre]

API Proposal

World:setKeyGenerator(function (state)
   local current = state
   state = state +1
   return string.format("%d", state), state
end, -2^53)

print(World.__hash.state) --prints: -2^53 (Don't modify this value manually)

Player:give("key") --Gives the built-in Key component to the Entity (which should belong to a World)

local key = player.key.value

local entity = World:getEntityByKey(key)

This means that the Key is an actual component that is serialized like any other, any Entity with said Component will get a unique Key (from the Key Generator) that can then be used (through Entity:get("key").value) to get it from the World with :getEntityByKey(key)

I think the most important detail is how state is handled. Basically :setKeyGenerator takes two values, the first is a generator function and the second is the initial state.

The generator function, takes the current state and returns two values, the unique key, and the next state.

The state is stored in the World, inside a hidden variable in World.__hash.state.

This state can then be serialized along with the Entities so that it can be recovered on deserialization.

Further Notes:

Changes in Component

This proposal requires a change in how Component:__initialize works, in which it should be able to receive the Entity as first argument to store it inside the Component for further access in the Populate function of said component.

This would allow the component to access both, the Entity and the World, which it needs to save the reference for that Key to that specific Entity.

Serialization

Serialization would require people to convert any Entity stored inside a Component to an Key and back in Deserialization. Other than that it shouldn't be that hard to recover.

Mismatched generators when deserializing.

One possible issue is recovering the state of the key generator from an old serialized state, and have a mismatch with the current key generator expected state. So a specific flag may be needed in deserialization to ignore said state, or upgrade the Keys if needed (probably not?).

[pablomayobre]

Regarding the last problem ("Mismatched generator when deserializing"), I consider it should be ignored entirely, since this problem already exists when you modify your Component definition, this should most likely be treated the same way.

There are some simple safety nets we can add, like not restore the state if an option is passed to World:deserialize but honestly I don't think we should worry about it too much.

[speakk]

Regarding the mismatched generator part, are you proposing that it's up to the user to handle key conflicts by hand?

For example in the following scenario: Deserializing older entities into an existing world, how do we handle that? (I can envision ways to deal with that by hand, but since the key stuff is now being handled by Concord, it would feel natural for this to be, as well?)

[pablomayobre]

Regarding the mismatched generator part, are you proposing that it's up to the user to handle key conflicts by hand?

For example in the following scenario: Deserializing older entities into an existing world, how do we handle that? (I can envision ways to deal with that by hand, but since the key stuff is now being handled by Concord, it would feel natural for this to be, as well?)

Yes and no, when you restore from a deserialized state, Concord can also deserialize the last generator state and start the generator from there. So in this case Entities generated after this wouldn't clash with the deserialized Entities.

However, if the World already had Entities we could have some issues, some possible fixes that can be implemented by the end user without problems are, changing the generator so that keys are prefixed with the current Unix timestamp (or similar), or just making sure no Entities exist in the World before deserialization.

The other issues is more about changes in the code from a previous deserialization, say I have version 1 of my game with a very generic key generator, and then in version 2 I change it for some complex UUIDs that require some custom state. If I recover the generator state from version 1 into version 2, the state will be invalid and the new generator will fail to generate a new key (Also all new keys will be very different to old ones, but that's not much of a problem)

This is a very normal issue in Components, say if in version 2 I change my position component into a transform and include scale and rotation. Or if I add a quad property to my sprite component. When deserializing a state from version 1, a definition for the position wouldn't be found, no Entity would have a transform component, and sprite wouldn't have the quad property.

[speakk]

Hmm yes, I can in theory get around this problem by adding the previous serialized entities first before I add new entities in. It would be to be able to be able to deserialize entities into an existing and populated World, but perhaps this is not so critical.

[pablomayobre]

Currently there is an issue with serialized keys, not working properly. An in depth investigation is needed.

@flamendless found this error, maybe he can give more information about it

[flamendless]

Currently there is an issue with serialized keys, not working properly. An in depth investigation is needed.

@flamendless found this error, maybe he can give more information about it

Done :+1: PR submitted and merged