feat(electrum): optimize merkle proof validation with batching
Replaces #1908, originally authored by @Keerthi421. Fixes #1891.
Description
This PR optimizes sync/full_scan performance by batching and caching key RPC calls to slash network round-trips and eliminate redundant work.
Key improvements:
- Gather all
blockchain.transaction.get_merklecalls into a singlebatch_callrequest. - Use
batch_script_get_historyinstead of many individualscript_get_historycalls. - Use
batch_block_headerto fetch all needed block headers in one call rather than repeatedly callingblock_header. - Introduce a cache of transaction anchors to skip re-validating already confirmed transactions.
Anchor Caching Performance Improvements
Results suggest a significant speed up with a warmed up cache. Tested on local Electrum server with:
$ cargo bench -p bdk_electrum --bench test_sync
Results before this PR (https://github.com/LagginTimes/bdk/tree/1957-master-branch):
sync_with_electrum time: [1.3702 s 1.3732 s 1.3852 s]
Results after this PR:
sync_with_electrum time: [851.31 ms 853.26 ms 856.23 ms]
Batch Call Performance Improvements
No persisted data was carried over between runs, so each test started with cold caches and measured only raw batching performance. Tested withexample_electrum out of https://github.com/LagginTimes/bdk/tree/example_electrum_timing with the following parameters:
$ example_electrum init "tr([62f3f3af/86'/1'/0']tpubDD4Kse29e47rSP5paSuNPhWnGMcdEDAuiG42LEd5yaRDN2CFApWiLTAzxQSLS7MpvxrpxvRJBVcjhVPRk7gec4iWfwvLrEhns1LA4h7i3c2/0/*)#cn4sudyq"
$ example_electrum scan tcp://signet-electrumx.wakiyamap.dev:50001
Results before this PR:
FULL_SCAN TIME: 8.145874476s
Results after this PR (using this PR's bdk_electrum_client.rs):
FULL_SCAN TIME: 2.594050112s
Changelog notice
- Add transaction anchor cache to prevent redundant network calls.
- Batch Merkle proof, script history, and header requests.
Checklists
All Submissions:
- [x] I've signed all my commits
- [x] I followed the contribution guidelines
- [x] I ran
cargo fmtandcargo clippybefore committing
New Features:
- [x] I've added tests for the new feature
- [x] I've added docs for the new feature
Bugfixes:
- [ ] This pull request breaks the existing API
- [x] I've added tests to reproduce the issue which are now passing
- [x] I'm linking the issue being fixed by this PR
@LagginTimes could you provide the benchmark results in the PR description and compare it to results before the changes in this PR?
Based on above benchmark results it looks like this change is 1s faster on sync, is that due to a small test size? Do we expect it to make more of a difference with wallets with many addresses?
@LagginTimes ~~can you provide the code you used to test with a remote electrum server (instead of the testenv)?~~
Edit: how about we just test with the example-cli with a pre-populated signet wallet?
I suggested writing benchmarks with the assumption that local io (against testenv) would be slower than allocating memory (collecting requests into vec before batch requesting), however that assumption seems incorrect.
These are my criterion results after benching this PR. 👍
sync_with_electrum time: [37.325 ms 38.220 ms 38.805 ms]
change: [-85.756% -85.556% -85.308%] (p = 0.00 < 0.05)
Performance has improved.
In 838c24762b8b03a34a64804b9b7a5e6626f03aad:
error: this file contains an unclosed delimiter
--> crates/electrum/src/bdk_electrum_client.rs:724:3
|
32 | impl<E: ElectrumApi> BdkElectrumClient<E> {
| - unclosed delimiter
...
504 | for &(txid, height, hash) in chunk {
| - this delimiter might not be properly closed...
...
547 | }
| - ...as it matches this but it has different indentation
...
724 | }