python-fpga-interchange icon indicating copy to clipboard operation
python-fpga-interchange copied to clipboard

Need canonical storage solution to ensure binary consistency

Open litghost opened this issue 4 years ago • 2 comments

Currently there are two sources of instability in the FPGA interchange capnp files:

  • strList order is unstable
  • Inconsistent compression output

This issue is to track adding logic to remove instability in the FPGA interchange capnp files. The goal is that the same data if canonical will generate the same file.

The strList order can be fixed by sorting the strList and re-numbering the ids. The inconsistent compression output can be solved by ensuring that the compress level between tools is consistent, and using a compression library that generates stable results.

litghost avatar Jan 21 '21 19:01 litghost

I don't think diffability is the major goal. More making sure that two files which have identical data contents will be identical.

IE If the diff tool returns that two files are identical, then the canonical storage version of the files should have identical checksums.

mithro avatar Jan 21 '21 20:01 mithro

I don't think diffability is the major goal. More making sure that two files which have identical data contents will be identical.

IE If the diff tool returns that two files are identical, then the canonical storage version of the files should have identical checksums.

Agreed, I'll update the description.

litghost avatar Jan 21 '21 21:01 litghost