nf-prov icon indicating copy to clipboard operation
nf-prov copied to clipboard

Add file checksums to BCO format

Open bentsherman opened this issue 9 months ago • 0 comments

The BCO format supports SHA1 checksums for URI objects. These checksums are particularly useful for intermediate output files since they are usually deleted once the run completes.

There is a Nextflow PR https://github.com/nextflow-io/nextflow/pull/3802 to save the MD5 checksum of each output file to the task cache to facilitate better provenance reporting. The MD5 checksum is advantageous because the big three cloud providers automatically compute the MD5 checksum of an object when it is uploaded, so Nextflow can re-use these checksums and save some time.

Unfortunate that BCO only allows for SHA1. We would have to compute SHA1 in addition to the MD5 checksums already being computed. Need to think on this more.

bentsherman avatar Sep 28 '23 03:09 bentsherman