kerchunk icon indicating copy to clipboard operation
kerchunk copied to clipboard

fletcher32 checksum

Open tedhabermann opened this issue 3 years ago • 3 comments

All datasets with fletcher32 == True are rejected by referenceMaker. This is the checksum supported by HDF5. Would it be possible to add it to nomcodecs? @ajelenak

tedhabermann avatar Jun 24 '21 19:06 tedhabermann

I think so. numcodecs already supports Adler32 which is similar to Flether32, I think.

ajelenak avatar Jun 24 '21 19:06 ajelenak

The code looks simple enough.

The question is, how is this value included in the target bytes? Is it just in metadata around the bytes? If yes, then we can already ignore the checksum and use the bytes; but implementing the function above would enable the (optional) validation.

martindurant avatar Jun 25 '21 12:06 martindurant

Example code of how to fetch the checksums: https://forum.hdfgroup.org/t/accessing-fletcher32-checksum/7299/3

Apparently it's just the last 4 bytes of the given block. So to ignore, we can just truncate.

martindurant avatar Jun 25 '21 12:06 martindurant