feat(core): multi-pubkey DB support

[onur-ozkan]

This pull request enables support for managing multiple public keys in mm2 databases. Initially, a single database instance is generated using the startup public key rmd160. With this update, the mm_ctx will manage multiple database instances by leveraging the public key from MmCoin::account_db_id if defined, or defaulting to the mm2 startup public key.

Completed

• [x] Introduced sql_connection_pool for managing all instances of shared and sqlite connections from one place
• [x] Introduced async_sql_connection_pool for managing all instances of async sqlite connections from one place
• [x] Refactored indexeddb driver implementation to allow multi pubkey support
• [x] Refactored SwapContext, MySwapStorage, MakerSwapStorage, TakerSwapStorage NftCtx, UtxoBlockHeaderStorage, Zcoin storages(web only), lp_ordermatch and other storage impl to support multipubkey
• [x] Implement database migrations upon new database instance creation or account activation using a custom public key.
• [x] swap_kick_start and orders_kick_start unfinished swaps/orders upon new database instance creation or account activation using a custom public key.

Checklists for next PR

• TODO: discuss and enhance get_public_key, get_public_key_hash, show_priv_key rpc for multi pubkey dbs in a another PR

[shamardy]

@onur-ozkan can't add you as a reviewer for this PR because you are the PR author, can you please review this PR?

[onur-ozkan]

Complex Todos:

• [x] Implement database migrations upon new database instance creation or account activation using a custom public key.
• [ ] kick_start unfinished swap upon new database instance creation or account activation using a custom public key.

Currently, without making a big refactoring, this can be achieved from the coin activation functions I believe.

[borngraced]

Complex Todos:

• [x] Implement database migrations upon new database instance creation or account activation using a custom public key.
• [ ] kick_start unfinished swap upon new database instance creation or account activation using a custom public key.

Currently, without making a big refactoring, this can be achieved from the coin activation functions I believe.

yep I accomplished task 1 by implementing a db migration watcher which keeps track of new db_id and run migrations/fix shared_db_dir if needed https://github.com/KomodoPlatform/komodo-defi-framework/pull/2093/commits/b370bec74069a71636c8695fc6c284f82fa3dce9

[shamardy]

@borngraced docker tests timeout after 90 minutes, my guess is that there is a test that uses the db that now runs forever after the current changes. can you please find the problem and try to fix it?

[smk762]

For QA reference, I'll list a few things here to be tested. Please add to the list (or tell me to remove) any items which are missing/not required.

With new HD wallet:

• Perform swap as maker / taker for UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
• Perform withdraw / deposit of the UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
• Test failed swap recovery for UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
• Test all NFT related functions.

DB migration tests: with old HD wallet (use master branch to generate some activity, then update to this branch):

• Perform swap as maker / taker for UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
• Perform withdraw / deposit of the UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
• Test failed swap recovery for UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
• Test all NFT related functions.

Regression testing

• With a legacy (nonHD) wallet, repeat the above tests.

Stress testing:

• Initiate as many simultaneous swaps as possible while also initiating as many other DB triggering methods at the same time (zhtlc activation, tx history, nft update etc).

With CLI:

• inspect all database tables to identify updates required for https://komodoplatform.com/en/docs/komodo-defi-framework/tutorials/query-the-mm2-database/
• Test seednode stats related methods

With Web:

• Test the following case: if the same seed is used in both Keplr and mm2, and a swap is initiated using Keplr, then the app was closed midway through the swap. Upon restarting, if the Tendermint coin is activated using mm2 seed and the address enabled for swaps is the same address that was used before in Keplr, the swap will complete successfully. and its inverse.
• Check tx history of tendermint assets to confirm tx history data is separated between inner and external accounts.

Outside the scope of testing

• get_public_key, get_public_key_hash, show_priv_key

Related issues for context

• https://github.com/KomodoPlatform/komodo-defi-framework/issues/2094
• https://github.com/KomodoPlatform/komodo-defi-framework/issues/1838