bdk icon indicating copy to clipboard operation
bdk copied to clipboard

feat(electrum): optimize merkle proof validation with batching

Open LagginTimes opened this issue 7 months ago • 5 comments

Replaces #1908, originally authored by @Keerthi421. Fixes #1891.

Description

This PR optimizes sync/full_scan performance by batching and caching key RPC calls to slash network round-trips and eliminate redundant work.

Key improvements:

  • Gather all blockchain.transaction.get_merkle calls into a single batch_call request.
  • Use batch_script_get_history instead of many individual script_get_history calls.
  • Use batch_block_header to fetch all needed block headers in one call rather than repeatedly calling block_header.
  • Introduce a cache of transaction anchors to skip re-validating already confirmed transactions.

Anchor Caching Performance Improvements

Results suggest a significant speed up with a warmed up cache. Tested on local Electrum server with:

$ cargo bench -p bdk_electrum --bench test_sync

Results before this PR (https://github.com/LagginTimes/bdk/tree/1957-master-branch):

sync_with_electrum      time:   [1.3702 s 1.3732 s 1.3852 s]

Results after this PR:

sync_with_electrum      time:   [851.31 ms 853.26 ms 856.23 ms]

Batch Call Performance Improvements

No persisted data was carried over between runs, so each test started with cold caches and measured only raw batching performance. Tested withexample_electrum out of https://github.com/LagginTimes/bdk/tree/example_electrum_timing with the following parameters:

$ example_electrum init "tr([62f3f3af/86'/1'/0']tpubDD4Kse29e47rSP5paSuNPhWnGMcdEDAuiG42LEd5yaRDN2CFApWiLTAzxQSLS7MpvxrpxvRJBVcjhVPRk7gec4iWfwvLrEhns1LA4h7i3c2/0/*)#cn4sudyq"
$ example_electrum scan tcp://signet-electrumx.wakiyamap.dev:50001

Results before this PR:

FULL_SCAN TIME: 8.145874476s

Results after this PR (using this PR's bdk_electrum_client.rs):

FULL_SCAN TIME: 2.594050112s

Changelog notice

  • Add transaction anchor cache to prevent redundant network calls.
  • Batch Merkle proof, script history, and header requests.

Checklists

All Submissions:

  • [x] I've signed all my commits
  • [x] I followed the contribution guidelines
  • [x] I ran cargo fmt and cargo clippy before committing

New Features:

  • [x] I've added tests for the new feature
  • [x] I've added docs for the new feature

Bugfixes:

  • [ ] This pull request breaks the existing API
  • [x] I've added tests to reproduce the issue which are now passing
  • [x] I'm linking the issue being fixed by this PR

LagginTimes avatar May 15 '25 19:05 LagginTimes

@LagginTimes could you provide the benchmark results in the PR description and compare it to results before the changes in this PR?

evanlinjin avatar May 26 '25 01:05 evanlinjin

Based on above benchmark results it looks like this change is 1s faster on sync, is that due to a small test size? Do we expect it to make more of a difference with wallets with many addresses?

notmandatory avatar May 26 '25 16:05 notmandatory

@LagginTimes ~~can you provide the code you used to test with a remote electrum server (instead of the testenv)?~~

Edit: how about we just test with the example-cli with a pre-populated signet wallet?

I suggested writing benchmarks with the assumption that local io (against testenv) would be slower than allocating memory (collecting requests into vec before batch requesting), however that assumption seems incorrect.

evanlinjin avatar Jun 04 '25 01:06 evanlinjin

These are my criterion results after benching this PR. 👍

sync_with_electrum      time:   [37.325 ms 38.220 ms 38.805 ms]                              
                        change: [-85.756% -85.556% -85.308%] (p = 0.00 < 0.05)
                        Performance has improved.

ValuedMammal avatar Jun 25 '25 13:06 ValuedMammal

In 838c24762b8b03a34a64804b9b7a5e6626f03aad:

error: this file contains an unclosed delimiter
   --> crates/electrum/src/bdk_electrum_client.rs:724:3
    |
32  | impl<E: ElectrumApi> BdkElectrumClient<E> {
    |                                           - unclosed delimiter
...
504 |             for &(txid, height, hash) in chunk {
    |                                                - this delimiter might not be properly closed...
...
547 |         }
    |         - ...as it matches this but it has different indentation
...
724 | }

ValuedMammal avatar Jun 25 '25 13:06 ValuedMammal