Pruned nodes

[RCasatta]

untestested initial attempt for https://github.com/RCasatta/blocks_iterator/issues/84

@AdamISZ would something like this be acceptable for your use case?

[AdamISZ]

Thanks very much, I'll try to test this out today.

[AdamISZ]

So on a first try, setting config.start_at_height to the first height as reported by getblockchainfo, this did not work for me, i.e. I got the same error as before ("reorder map too large" etc), but then on reading the code more carefully I realise I don't understand it.

start_at_hash is a private field in the config. Then when you call start_at_hash() as a function, it matches on this value, but it was never set anywhere I think? Which means we just go back to the default genesis_block()?

Shoul start_at_hash just be public? I can try that.

(A related question, while I can use the syntax I described in my "hack", in Rust code, to specify a BlockHash from a hex string, I don't know how I would do that in a config file .. I guess again it can be just converted from hex, but I'm not sure.)

[AdamISZ]

I did a lot of experimentation today, eventually realizing (I think?) that there is a more fundamental problem here than I had first imagined. Even worse, the problem I discovered did not show itself when I first tried my "hack" solution and now, I cannot reproduce my original result (I don't know what changed; I'm using the same blocks database and I get the same prune height value of 781485 as I did then).

In trying to debug i discovered this; this is the result of the first call to exist_and_has_followers:

Starting with next hash: 0x00000000000000000004403bd59d5cfd47a2c225f85f238c2b0287f996017a24
And self.blocks: {
    0x00000000000000000004403bd59d5cfd47a2c225f85f238c2b0287f996017a24: FsBlock {
        file: Mutex {
            data: File {
                fd: 4,
                path: "/home/ekaggata/.bitcoin/blocks/blk03484.dat",
                read: true,
                write: false,
            },
            poisoned: false,
            ..
        },
        start: 8,
        end: 1454170,
        hash: 0x00000000000000000004403bd59d5cfd47a2c225f85f238c2b0287f996017a24,
        prev: 0x000000000000000000049ec581c654ef55e56edae782cfcaafdde7ea325ec632,
        next: [],
    },
}

(I deliberately hardcoded the input hash function as the one for this block) ... this block hash corresponds to height 781479, i.e. 6 blocks earlier, but I notice that "next" is [] (does that mean it's not on the main chain?). I'm rather clueless as to what's happening at this point.

I realize this description is not nearly complete enough for you to figure it out, but just recording it for now. I don't think I can make further progress (which is fine, I have no urgent need for this, I'm just curious as to what's happening).

[RCasatta]

does that mean it's not on the main chain?

No, the next field may be populated later when other blocks are read. There is no guarantee that the blocks found in the .dat files are in order, to avoid during a double pass this logic is applied.

I need to create a test vector of the blocks directory when pruned so that I can work on this with a test case...