docs: detail on integrity checks

[udf2457]

trafficstars

Looking through the codebase, it looks like lightningstream does not use the Content-MD5 header in S3-compatible APIs in order to ensure end-to-end integrity of blobs uploaded.

It would be nice to have a couple of paragraphs in the docs as to how lightningstream approaches this.

[wojas]

Lightning Stream uses https://github.com/PowerDNS/simpleblob for object storage, which uses https://github.com/minio/minio-go for S3 storage.

[udf2457]

Sure, I looked briefly through that code yesterday. However you need to explicitly set the MD5 option in the minio-go struct when calling it and - as far as I can tell after a quick glance - simpleblob is neither calculating the MD5 or calling minio-go with the MD5 option.

Sending Content-MD5 is highly recommended, because otherwise the receiving S3 service has no way of ensuring integrity. With Content-MD5, the S3 service caluclates MD5 and compares it against the MD5 provided in Content-MD5, and so you therefore have a high level of confidence that what you uploaded is what was stored.

[wojas]

Thanks, I added PowerDNS/simpleblob#49 for this and added it to the next patch release milestone.

[udf2457]

Thanks @wojas !