Update bdk_bitcoind_rpc module to work with a pruned bitcoind node

[notmandatory]

Summer of Bitcoin Project Proposal

Description

The current bdk_bitcoind_rpc module requires full access to transaction history using a non-pruned node (a node storing the full blockchain, which at the time of writing is over 400GB).

However, even without a wallet's full transaction history it is still possible to compute the wallet balance and be able to spend with only a pruned bitcoind node using the RPC method scantxoutset. This project will update the existing bdk_bitcoind_rpc module to work with a pruned bitcoind node.

Expected Outcomes

• Updatefull_scan to use the scantxoutset if the bitcoind node is pruned.
• Create tests to confirm full_scan works with a pruned node.
• Update rpc examples and test with a pruned bitcoind node.

Resources

• bitcoin core node setup.
• bitcoin rpc commands.
• issue 895.
• https://developer.bitcoin.org/reference/rpc/scantxoutset.html
• https://github.com/bitcoindevkit/bdk/pull/1041
• https://github.com/bitcoindevkit/bdk/pull/1172

Skills Required

• Experience with git. Guide
• Experience with rust. First seven chapters of the book
• Able to setup a local core node in regtest and testnet mode.
• Familiarity with bitcoind RPC commands.

Mentor(s)
@notmandatory

Difficulty
Hard

Competency Test

• Install rust, compile and run all bdk examples and tests.
• Setup a local Bitcoin Core pruned node daemon in regtest mode.
• Make a wallet with example_bitcoind_rpc_polling example wallet and receive and send regtest bitcoin.

[rustaceanrob]

Mind if I work on this? I'm not SOB but I am doing a similar program haha

[c0llinx]

Is this still open? Would like to give it a shot

[LLFourn]

Not sure if our architecture actually supports this kind of usage. The problem is how do you figure out when an output has been spent? The only way is that it is no longer returned from the scantxoutset output. But in bdk_chain we tell it outputs are spent by providing a full transaction that spent it. We can't get that here.

It can be used to find your current balance, but can't be used in a way where you persist anything. So the workflow would be to just scantxout then get the outputs, create a transaction from them and spend them. You could .insert_txout them into a temporary Wallet and create transactions from there. I think that's the only way to get value from bdk with this atm.

[star-gazer111]

Hey @notmandatory the link you shared for rpc commands (https://developer.bitcoin.org/reference/rpc/) is somehow not working.

[star-gazer111]

Hey @notmandatory the link you shared for rpc commands (https://developer.bitcoin.org/reference/rpc/) is somehow not working.

I think this can be a good alternative resource for the same.

[jaoleal]

@star-gazer111 https://chainquery.com/bitcoin-cli i prefer to use this one, its kinda better.

[notmandatory]

@LLFourn what if we use the RPC syncing as it is now but with a bitcoind node set to "manual" pruning? then we just have to add functionality to call the "pruneblockchain" command after syncing to remove old (maybe older than 1000 deep or so) already ingested blocks.

[LLFourn]

@notmandatory yeah cool I didn't know about this setting but that also works. I feel like the usage is a little niche:

1. You don't want to run an archival node
2. You can guarantee your wallet will come online often enough to drain the blocks and prune.
3. You only have one wallet system.

Certainly not an unlikely situation within a small Bitcoin startup. Also can exist in user facing software that actually embeds a pruned node.

[notmandatory]

Removing from 1.1 milestone since no PR ready.

[luisschwab]

Still waiting for corepc to mature a bit more so we can tackle the dependency replacement. We need to discuss what would be done about the async part.

[luisschwab]

@oleonardolima can you move this to bdk?

[AdamISZ]

Hi, just thought I'd note down my own initial experience: this seems to be a substantial problem, given the negatives of using a wallet without one's own node, and the newer negative of running a full (archival) node given how close we're getting to the 1TB magic number.

In trying to set up a wallet myself I hit the snag: newly created wallet (using RPC) must be sync-ed but you cannot tell it to not check every block from genesis afaict. So stuff like:

let mut emitter = Emitter::new(
        &rpc_client,
        wallet_tip.clone(),
        wallet_tip.height(),
        NO_EXPECTED_MEMPOOL_TXS,
    );

... with wallet_tip defined as a checkpoint using a recent block, will not help at all, as it won't be able to connect blocks with apply_block_connected_to (I believe that's what I should be using? I'm following the book of bdk material primarily). And as far as I can tell it would be either highly non-trivial or a fool's errand to try to make some kind of minor patch to allow this.

Yet it seems like it should be possible: sync could mean: only start scanning blocks from block N. (You could either trust the source or not when it comes to headers, doesn't seem bad either way)

I guess my main point relates to that last sentence: wouldn't that be the logical thing anyway, even forgetting the concept of pruning? i.e. let the developer/consumer of this library make an explicit choice about wallet birthday (albeit, for obvious reasons, you may sometimes not be able to)?

(And my secondary point is, if I am missing something that would allow this please let me know!)

[notmandatory]

I suspect the way to start syncing from a birthday block would be to use Wallet::apply_block_connected_to() with params birthday_block, birthday_block_height, genesis_block_id. But I haven't tried it so I could be missing something.

This issue hasn't gotten much attention since the focus for "light client" block syncing has moved to @rustaceanrob 's work with https://github.com/2140-dev/kyoto. For most users CBFs should be a superior approach.

But still leaving this issue open for any summer of bitcoin students, or anyone else who wants to try taking it on.

[rustaceanrob]

Bitcoin Core had a few big PRs merge since the last release, and I think there might be some higher value projects for SoB. One would be to use the newly-merged kernel API, which exposes validation and persistence via a C header. Bindings for this are available here, and I expect them to move into an organizational repository soon. A possible project could be to read the blocks from disk and sync a BDK wallet.

A mining interface has also been exposed via the multiprocess project. There are some interesting branches that may be used to interact with bitcoind over Capn' Proto, but I would consider that project more advanced.

[AdamISZ]

I suspect the way to start syncing from a birthday block would be to use Wallet::apply_block_connected_to() with params birthday_block, birthday_block_height, genesis_block_id. But I haven't tried it so I could be missing something.

Yes, I was using that method (as mentioned, and as suggested in the book of bdk example), the RPC call errors with "Block not available (pruned data)".

This issue hasn't gotten much attention since the focus for "light client" block syncing has moved to @rustaceanrob 's work with https://github.com/2140-dev/kyoto. For most users CBFs should be a superior approach.

Right, I had seen that but indeed perhaps I should focus more on it.

I guess I'm just really curious to know if it's true that the RPC based wallet requires actually parsing/scanning every single block since genesis, or not, in the codebase as-is; that is what my code (which again, is just copied in structure from the book of bdk example, and uses apply_block_connected_to in a next_block() iteration of the Emitter) apparently does. That seems wildly impractical, even for a non-pruned node; but it's very likely I'm simply missing something. Sorry for raising this in this Issue, it's only sort of half-related to the actual proposed update.

[luisschwab]

We need the scantxoutset RPC to get this done. Once the new bitcoind RPC client becomes production ready I can tackle this issue.

[notmandatory]

Another data-point, using a modified unit test from bitcoindevkit/bdk_wallet#336 I was able to apply a new block with a gap between it and the genesis block and got the expected result. So it should be possible with the bdk_bitcoind_rpc crate to start syncing from a birthday block instead of every block back to genesis.

bdk_wallet/tests/wallet_event.rs

#[test]
fn test_apply_birthday_new_block_event() {
    let (desc, change_desc) = get_test_wpkh_and_change_desc();
    let params = Wallet::create(desc.to_string(), change_desc.to_string());

    let mut wallet = params
        .network(Network::Regtest)
        .create_wallet_no_persist()
        .expect("descriptors must be valid");

    // apply birthday block
    let genesis = BlockId {
        height: 0,
        hash: wallet.local_chain().genesis_hash(),
    };
    let block1 = test_block(
        genesis.hash,
        1000,
        vec![],
    );
    let events = wallet.apply_block_connected_to_events(&block1, 1000, genesis).unwrap();
    assert_eq!(events.len(), 1);
    dbg!(&events);
    assert!(
        matches!(events[0], WalletEvent::ChainTipChanged { old_tip, new_tip } if old_tip ==
genesis && new_tip == (1000, block1.block_hash()).into())
    );
}