Disabling ledger needs to be marked in Database

[Cmdv]

When setting the configuration ledger: disabled it seems that the following is run every time db sync is restarted:

update redeemer set fee = null
delete from reward
delete from epoch_stake
delete from ada_pots
delete from epoch_param

This causes a long pause even though it's already been ran. There are about ~45million redeemer entries (mainnet) so even if they are set to null, for the db to rerun the update will cause a delay.

We can mitigate this by simple adding a value to extra_migration I'm unsure if we can re-enable ledger once it has been disabled so we wouldn't need to deal with removing it.

[rdlrt]

In general would be nice if config settings itself are saved in database, which may be usable to easily verify current runtime config - be it internally for dbsync design/development or externally for users of dbsync.

A side benefit of this is that it also helps to validate the snapshot v/s config of dbsync.

[Cmdv]

@rdlrt interesting, I like the ideas. We'd have to think of scenarios where configs are changed back and forth by the user and how that might effect snapshots. Also so configs are "permanent" as in can't be put back.

Should add that internally we always keep a representation of the config for various tasks as well as check anything previous in extra_migration table (which would probably be incorrectly named for this feature).

Lastly I don't think we could support generating a multitude of snapshot for multiple configuration settings if that's something that would also be needed. 🤔

[rdlrt]

@rdlrt interesting, I like the ideas. We'd have to think of scenarios where configs are changed back and forth by the user and how that might effect snapshots. Also so configs are "permanent" as in can't be put back.

All the more reason I feel to maintain configs , if someone has changed the config options and dbsync detects the change at startup it should add a record in the config table - alongwith latest block where config was changed (so if changed can be tracked and find faults easier). Also helps if someone went through a config change path that shouldnt be supported

Lastly I don't think we could support generating a multitude of snapshot for multiple configuration settings if that's something that would also be needed. 🤔

Oh - absolutely, we dont expect snapshots to be provisioned by IO, we are maintaining a flavour (with consumed enabled) on Koios releases, and with the recent merged PRs hope to start providing lite (i.e. with prune and ledger disabled) version as well. Enabling two additional flavours of snapshots to be maintained for each network. So when receiving a support request, such a config table would certainly help out.

[kderme]

Saving the config e.g. as json in the db is quite reasonable, but the config format may change. This would force us to support backwards compatibility for it, which is not very desirable, especially since parts of the modularity are still experimental.

We could save the config just for verifications reasons by the user. This could still cause minor inconsistencies if the config changes, eg duplucate entries for equivalent configs, however it wouldn't be a huge deal.

Right now we only save the necessary parts in the extra_migration table. For this specific issue, we could either use this table or figure out if the migration is necessary by querying other tables. For examples, if there are no ada_pots or epoch_param, then this migration doesn't need to run at all.

[rdlrt]

This could still cause minor inconsistencies if the config changes, eg duplucate entries for equivalent configs, however it wouldn't be a huge deal.

The idea is to save config section and block height whenever a migration is required, so that it is trackable when an instance changed historically, also - the content of config changing shouldnt have impact if the JSON section with insert related options are being added (essentially just a field with values). The primary consumer for this information is restore snapshot scripts, which we can modify to verify config matches with snapshot