electrumx
electrumx copied to clipboard
spesmilo
implement protocol 2.0; change db schema
This implements Electrum Protocol 2.0, see https://github.com/spesmilo/electrumx/pull/90 (formerly named version 1.5). Supersedes https://github.com/spesmilo/electrumx/pull/80 (see existing comments there). Includes https://github.com/spesmilo/electrumx/pull/109.
Note: this is ~done; but we probably won't merge it until also implementing the protocol changes in the client, to avoid having to change the protocol again due to some unforeseen issue (though I don't expect issues). I'll implement the changes in Electrum after the 4.1 release.
UPDATE: performance was better on r7g.2xlarge (32GB RAM). rocksdb driver that was used might also have higher RAM reqs than the default leveldb
New schema looks good. At first attempt at syncing with this, performance isn't great, unfortunately. On AWS' r6g.large instance (dual-core ARM64 w/ SHA extensions, 16GB RAM) with pypy and rocksdb, current spesmilo:master syncs in under 24 hours, whereas this branch is estimated to take about a week. I don't know why CPU isn't able to max out, nor is disk throughput. It might be some very large pages being used in memory, hard to tell.
Anything else you want me to try, e.g. on the same compute configuration, let me know.
What value of CACHE_MB did you test with? If default, it might be worth trying with 2x that.
(The cache stores a lot more data for the history dbs now:
see history.py/unflushed_memsize https://github.com/spesmilo/electrumx/pull/101/files#diff-55c36e9a34bb1ea334f60433845bca3c631058fdf83cbe6dc662a531d8a1b35fR211)
hi, what is the expected timeline to see protocol 1.5 merged in electrumx and implemented in electrum? i have a few wallets which can't load because of its very long transaction history
Re my comments above, I was able to get sync within a couple days testing this on 32GB of RAM instead of 16GB. I don't know if Linux decided to do small-but-frequent swapping or if RockDB is self-limiting, but either way it looks like the new schema just has larger system requirements.
I'll implement the changes in Electrum after the 4.1 release.
@SomberNight Electrum 4.2.1 is out; are there any plans regarding when Protocol 1.5 will be added to Electrum?
For client devs wanting to test the new protocol, here are electrumx servers:
Bitcoin testnet - db2.electrum.justinarthur.com:51001
Bitcoin mainnet - db2.electrum.justinarthur.com:50001
Both use TLS. Use mainnet server with caution… this is new tech being tested.
Metrics and logs for both servers: https://db2.electrum.justinarthur.com/
use electrum_dev as the username and password.
It's not automatically pulling on branch changes so ping me on IRC if any of you need me to pull in new changes, a new branch, or start a new sync. I'm using my cloud vendor's spot market, so servers are subject to occasional interruption if my cloud vendor's customers increase their regularly contracted usage.
UPDATE: Bitcoin mainnet sync has completed. Took 3d10m
I'm shutting down the test servers to save money. If anyone is actively developing on Electrum or similar stratum client for protocol 1.5 and needs infrastructure to test against, just ping me on here or as jarthur in #electrum on Libera.
