flow-go

flow-go copied to clipboard

Problem

CRC-32 is too small for 170GB checkpoint files (as noted in PR #1944 when file size was closer to 60GB).

The Proposed Solution

I suggested replacing CRC-32 and adding concurrency in a comment in checkpointer.go:

https://github.com/onflow/flow-go/blob/c26a026acecefd72be14561deb43f210b04fdde3/ledger/complete/wal/checkpointer.go#L294-L295

We can replace CRC-32 with a 256-bit hash and use concurrency to reduce loss of speed. Go provides SHA-256 and SHA-512/256. BLAKE2s and BLAKE2 (512-bit) are popular and reasonably fast on most modern CPUs.

BLAKE3 is very fast but its implementation would be from 3rd party and should be extensively tested with huge file sizes.

If cryptographic hashes are too slow, then noncryptographic hashes with smaller digest size is another possibility. However, best practice recommendations for file verification since 2012 is to use a 256-bit or larger hash such as SHA-2 or SHA3.

Text of PR #1944 has more context. Among other things, it mentioned possibility of replacing CRC-32 with BLAKE2 or BLAKE2s after adding concurrency.