atomicDEX-API icon indicating copy to clipboard operation
atomicDEX-API copied to clipboard

make mm2 connect only to a subset of passed electrums

Open cipig opened this issue 1 year ago • 0 comments

The situation atm is as follows:

  • electrum servers are configured in coins repo in the files https://github.com/KomodoPlatform/coins/tree/master/electrums
  • the scripts from https://github.com/KomodoPlatform/coins/tree/master/utils are run once per day and they produce different configs that can later be used by the apps, eg https://github.com/KomodoPlatform/coins/blob/master/utils/coins_config_tcp.json ... this file only contains max. 3 servers, no matter how many are in the initial config... the script simply tries to connect to the servers, no other checks (eg height) are done
  • the servers from the coins_config_tcp.json are passed to the electrum call by the apps

This approach has some drawbacks:

  • if one or more of the servers are lagging behind the chain and mm2 uses them, it will show a wrong balance and swaps will fail too
  • if the 3 servers selected by the script in coins repo are later failing, the coin will not work any more, even though the initial config has eg 6 servers and the others are still working
  • if the SSL certificates of the 3 selected servers expire, the coin can't be enabled any more, even though TCP would still work (assuming TCP ports are also configured in the initial config, like it is with all servers run by us, see eg KMD)

I propose to make mm2 do all the filtering to which servers it connects and to which it doesn't, so if i pass 6 electrum servers to the electrum call, it should choose eg 2 or 3 good working ones and connect to them. The filtering should include the following:

  • don't connect to servers that are lagging behind the chain... eg choose 2 with the highest height... if there are only 2 and they differ, then connect only to one... there should be a tolerance though (optimal would be if the tolerance is not a fixed number of blocks, but dependent on the block time of the chain, so for BTC 3 blocks is a lot while for DGB 3 blocks is still acceptable... coins file contains avg_blocktime in seconds, which can be used for this)
  • don't connect to the same server twice, even if it is passed twice to the electrum call.. this is done often, since config in coins repo contains both TCP and SSL ports for many servers, even though it is the same server... maybe we can also go down to the IP level, so if 2 hosts in DNS point to the same IP, mm2 connects only once to one of them... this will also reduce the number of file descriptors used and help Mobile since it only support 128 open files afaik
  • optionally we can also filter based on connection time, even though i guess that it is not that important since most time is spent with processing the request not with the sending/receiving the data itself... maybe define "connection time" as the time needed to process a certain call, since that also includes the speed of the actual server and not only the time needed to connect to it... i propose to use blockchain.headers.subscribe for this test... it also returns the latest height, so the data can be used for the filtering based on height

Left overs from #1966:

  • [ ] Randomize the primary servers (assuming here primary servers are the first x servers where x is specified in the enable request). This helps with load balancing: https://github.com/KomodoPlatform/komodo-defi-framework/pull/1966#discussion_r1755217466
  • [ ] Allow enabling the coin without requiring any connections to be established (disconnected coin): https://github.com/KomodoPlatform/komodo-defi-framework/pull/1966#discussion_r1755727567
  • [ ] Add & remove electrum servers in runtime (after enabling the coin): https://github.com/KomodoPlatform/komodo-defi-framework/pull/1966#discussion_r1755732444
  • [ ] RPC for the list of currently connected electrums: https://github.com/KomodoPlatform/komodo-defi-framework/pull/1966#pullrequestreview-2298035702
  • [ ] RPC for electrum server version checking: https://github.com/KomodoPlatform/komodo-defi-framework/pull/1966#discussion_r1755210476
  • [ ] Internal:
    • [ ] Make ElectrumConnection so it can send the defined electrum requests bypassing the owner client. To eliminate the circular dependency between the two objects: https://github.com/KomodoPlatform/komodo-defi-framework/pull/1966#discussion_r1755725034
    • [ ] Defer the on_connected signal till the version is checked.
    • [ ] Add a new signal to the ElectrumConnection to disconnect itself (so don't allow others to disconnect it). This helps with not having to call the event_handlers ourselves since the electrum connection will call them when it disconnects.
    • [ ] Make sure we don't disconnect from anywhere (any thread) unless the connection errored. We need to have some mechanism to count the number of threads using the connection and only disconnect when all of them don't need it anymore (or if the connection errored).

cipig avatar May 06 '24 15:05 cipig