Does caching with clair work at the moment?

[kcalvinalvin]

trafficstars

I'm not sure what the development status is right now for the caching part. Is it still in development or has it run before and you were working things out?

[adiabat]

Clair was an attempt to implement clairvoyant caching (Bélády's algorithm) to tell the pollard struct what to keep in memory and what to forget about (which would need to be put back in later with proofs).

It's optimal for a given number of leaves, which unfortunately is not a fixed amount of memory, as nearby branches are small and distant branches are big. Getting it to target a fixed memory size seemed quite complex and I didn't try that.

Even with variable branch sizes, it really feels like the clairvoyant algorithm should help a lot; with fixed look-ahead caching, the first 300K blocks of mainnet use a lot of proofs, but the utxo set is so small the whole forest should fit in RAM. So I was hoping for a significant reduction in IBD download with the clairvoyant caching.

I got it to the point where I could test it with ibdsim (it added yet another step of parsing through the whole blockchain to generate a cache schedule) but it only made a 3% reduction in download size. Much less than I expected, but maybe look-ahead is close to optimal. Or I messed something up with the clair code.

I needed to get the paper out so I went with the look-ahead numbers and gave up on clair for now. It is still puzzling that it didn't help more, so it might be interesting to look at again, but there's lots of other improvements I'm looking at first.

Happy to answer questions about it though! I think if you go back in the git history you can get a version where clair works. (and apparently ibdsim didn't work after that!)