Migration of models when schema changes

[glihm]

trafficstars

Currently, Torii doesn't support model changes in the database. However, model changes are now backward compatible at the storage level in dojo-core, and a game developer may change a model during the lifetime of a game.

What's Torii needs is at first a simple strategy that is defined from the schema changes of the model, in order to update the tables in the database as necessary.

• [ ] New field added
• [ ] Field removed
• [ ] Field type has changed (we may not support this one, or only simple changes with compatible data types).

[zarah-s]

@glihm , I'd love to have more details on what's expected here

[glihm]

@glihm , I'd love to have more details on what's expected here

For sure, the thing is that in Dojo, Torii (the indexer) is reading information from the world that is on-chain by gathering events. Once the events are gathered, Torii keeps data in its own database.

Currently, it's a SQLite database.

So when a model is registered, it has a pre-defined layout (members of the string, and the types for each of them). And currently, if you change the type of a member inside a model, Torii doesn't support to actually migrate the data to support the new type.

However, some types changes may be totally transparent. If you have a u8 that is then a u32, everything should be easy to migrate.

struct MyModel {
    #[key]
    player: ContractAddress,
    data: u8,
}

Ideally, Torii should support migrating to:

struct MyModel {
    #[key]
    player: ContractAddress,
    data: u32,
}

So we may have a first iteration where only simple types are supported. Torii processes the events here: https://github.com/dojoengine/dojo/tree/main/crates/torii/core/src/processors and more precisely the register of models here: https://github.com/dojoengine/dojo/blob/main/crates/torii/core/src/processors/register_model.rs.

The models are registered here: https://github.com/dojoengine/dojo/blob/b04c0d789977ecce3503559540dc4bfd62dc45e8/crates/torii/core/src/sql.rs#L97

[zarah-s]

Thanks for this info, can I hop on this? @glihm

[zarah-s]

Hi @glihm, So I browsed thoroughly through some key functions and discovered that the register_model function handles both the insert and update(on conflict) of a model.

This query handles it:

INSERT INTO models (id, name, class_hash, contract_address, layout, packed_size, \
unpacked_size, executed_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(id) DO \
UPDATE SET contract_address=EXCLUDED.contract_address, \
class_hash=EXCLUDED.class_hash, layout=EXCLUDED.layout, \
packed_size=EXCLUDED.packed_size, unpacked_size=EXCLUDED.unpacked_size, \
executed_at=EXCLUDED.executed_at RETURNING *

So when a member-type changes, the register_model function can still be called upon for the type-migration...

However, the contract_address will change if a new address is passed down the function during the migration (which I think is a drawback)

So I want to be sure I'm on the right track. If I am, then I seek your permission not to allow contract_address to change whenever the UPDATE part of the query is called upon

Let me know if I'm on the right track or not

[glihm]

Hi @glihm, So I browsed thoroughly through some key functions and discovered that the register_model function handles both the insert and update(on conflict) of a model.

This query handles it:

INSERT INTO models (id, name, class_hash, contract_address, layout, packed_size, \
unpacked_size, executed_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(id) DO \
UPDATE SET contract_address=EXCLUDED.contract_address, \
class_hash=EXCLUDED.class_hash, layout=EXCLUDED.layout, \
packed_size=EXCLUDED.packed_size, unpacked_size=EXCLUDED.unpacked_size, \
executed_at=EXCLUDED.executed_at RETURNING *

So when a member-type changes, the register_model function can still be called upon for the type-migration...

However, the contract_address will change if a new address is passed down the function during the migration (which I think is a drawback)

So I want to be sure I'm on the right track. If I am, then I seek your permission not to allow contract_address to change whenever the UPDATE part of the query is called upon

Let me know if I'm on the right track or not

Sorry for the delay here. Every time a model changes, the model will be re-deployed, so the address will change that's a good point as we need to support that. Also, currently the name is the only way to identify the model's contract. In a coming upgrade for dojo 1.0, we use a tag to identify a model namespace:name, which is a unique combination. Hence, the namespace:name tag could be used for more control on the database records.

Currently there's a wip by @Larkooo to integrate the namespaces into Torii (PR will be open against the feat/namespace branch). In the meantime, you could propose something on the current main using the name as identifier. To then easily be updated to use the tag.

Does it make sense to you?

[zarah-s]

Yea... This makes total sense now...

Final questions

• The namespace:name will be under the column name in the model table right?
• Is the namespace a placeholder or is it actually "namespace"... Let's say for instance, my model name is "myModel", will it be something like namespace:myModel? or {{{namespace}}}:myModel

[glihm]

Sorry for the delay here, how are you doing on that?

The namespace:name will be under the column name in the model table right?

The tag is now namespace-name, it's on main and you can check it out. The table name is the tag.

Is the namespace a placeholder or is it actually "namespace"... Let's say for instance, my model name is "myModel", will it be something like namespace:myModel? or {{{namespace}}}:myModel

The namespace is a placeholder yeah. 👍

[glihm]

@zarah-s any update on your end?

[glihm]

This is being worked by @Larkooo on #2637