EnterpriseDB
Barman 1.36 -> Noobaa S3 backup failure to upload data
When using Barman 3.16.1 to upload to a Noobaa S3 bucket, the backup fails with:
ERROR: Backup failed uploading data (An error occurred (InvalidDigest) when calling the UploadPart operation: The Content-MD5 you specified is not valid.) ERROR: Backup failed uploading backup.info file (An error occurred (InvalidDigest) when calling the PutObject operation: The Content-MD5 you specified is not valid.)
Backups using an older Barman version (3.11.1) work fine. I appreciate there's a bit of a gap between these two versions. But looking at the 1.36.0 release notes, I see the pin on the older version of boto3 (that produced these headers by default) was removed so I think it's likely that this is the case:
I see in the release notes you mention Minio, and in #1124 an additional compatibility issue was fixed for Cloudian HyperStore. My assumption is that Noobaa is another S3 implementation which requires the old behaviour for the particular operations that Barman is performing, while HyperStore/Minio do not. Which seems a bit strange to me but I don't know enough about the details of S3 to form a firm opinion.
I'm also going to work this from the Noobaa side once I've reproduced it on a current version (I'm currently using Noobaa 5.12.13-01521aa which is a bit old), but I think it's useful to file an issue here to make you aware & provide visibility for anyone else who might be searching the web for the above error messages in relation to cnpg/Barman.
Hi,
The difference between the fix we provided in 3.16.2 and your problem is that the former is related with the deletion of objects, while the latter is related with uploading objects.
In any case, thank you for letting us know, and do tell us how your test go with newer versions of Noobaa.
We have an internal task to modify the way we check for this "faulty software" object storage, which will likely fix this issue. We don't know if this will be in 3.17 or in a patch release for 3.16, but we are planning for the next release.
Thanks. The short term workaround of rebuilding the cnpg images with a downgraded boto3 version is working fine. And once I have cnpg with current boto3 tested against the latest Noobaa S3 I will report back.