Migration for address table of existing data

[Cmdv]

It was asked on our discord

Does turning on use_address_table config option require full resync?

The current answer is yes but we could provide some form of migration if the database is already populated.

Investigate if this is possible to do as part of the schema variation when using use_address_table config

[kderme]

One possible way to implement this is to follow steps

• Create a new table tx_out_use_address with the same schema as tx_out with enabled use_address_table
• Query pages of the tx_out table. For each page, extract the address related fields and insert them into the address table and use the returned address.id to create a use_address_table page.
• The (address.id, address) pairs should be cached. Since db-sync won't have parsed the ledger state yet, meaning enough available memory, it could cache all of the entries without evicting any.
• When all pages are complete, drop the tx_out table and rename tx_out_use_address to tx_out
• Clean the address pairs from the cache.
• Start db-sync

It's not trivial at all and the migration will still take some time.

[rdlrt]

Wouldnt this anyways touch each and every tx_out entry, making timeline similar (as in maybe 50-60%) to synching from scratch

[Cmdv]

yeah I've mad a quick attempt but I think you are right that it might be just as quick to sync from scratch. What do you think @kderme ?

[Cmdv]

I'm going to close this given the recent perf improvements to do this is going to be quicker starting from genesis/scratch.