hostd icon indicating copy to clipboard operation
hostd copied to clipboard

Cache sector roots for proof building

Open ChrisSchinnerl opened this issue 1 week ago • 3 comments

As the amount of data stored by indexers on the network goes up, so will the number of integrity checks they perform. Right now each integrity check reads a full sector from disk to build the merkle proof for the returned data.

If every sector is checked every 2 weeks, a host storing ~5TB of data will have 1 sector (4MiB) read from disk per second. That is a lot of constant I/O for a not particularly large host. We should come up with a plan to reduce that wear on a host's disks.

A potentially quite efficient solution would be to hash 4KiB subtrees of the stored data. Leading to 32KiB of cached roots per 4MiB of stored data. An approximately 0.08% overhead. Going below 4KiB wouldn't give us any benefits in terms of I/O anyway since that's usually the atomic read/write size of modern disks.

That said, 0.08% is still almost 9GiB of overhead per 1TiB of stored data. So it won't be feasible to store this in RAM.

ChrisSchinnerl avatar Nov 20 '25 12:11 ChrisSchinnerl