How can I specify `multipart_chunksize` in `barman-cloud-backup`

[atsu0127]

Hi, I am using barman-cloud-backup to back up to S3-compatible storage. In doing so, I need to specify the chunk size when multipart uploading.

So I was wondering if I could use TransferConfig, is there any way to specify this in barman-cloud-backup?

• about TransferConfig…https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html

Thanks,

[mikewallace1979]

Hi,

Barman uses boto3 multipart upload directly rather than using the managed transfers so TransferConfig won't have any effect currently.

The chunk size is determined automatically based on some hardcoded defaults (themselves based on S3 multipart upload restrictions) and the value of --max-archive-size as follows:

    self.chunk_size = 2 * int(
        max_archive_size / self.cloud_interface.MAX_CHUNKS_PER_FILE
    )

With the hardcoded values and the default --max-archive-size of 100GB, the chunk size for an S3 multipart upload will be 2 * (107374182400 / 10000) or 21474836 bytes.

For a smaller chunk size you could reduce the value of --max-archive-size (or increase it for a larger chunk size, though there is a hard limit of 1TB for the maximum archive size in S3). Note that reducing --max-archive-size will cause the backup to be spread across multiple objects in the object store, e.g. if a backup was 100GB and stored in a single data.tar file, then a --max-archive-size of 10GB would leave you with nine additional objects named data_0001.tar, data_0002.tar etc.

Does this help at all?

[Aransh]

Hey @mikewallace1979 Responding to this issue as I'm not 100% sure if this is related to my issue or not. I am also using an s3-compatible service, which cloudnative-pg (Specifically Linode Object storage, which is documented as supported by CNPG). I've been testing backing up and restoring back up, backup resource always shows as successful, but once I try to bootstrap from backup I get an error for missing WAL files. Looking at the logs from the replica that backup resource used, I can see this error:

{"level":"info","ts":"2024-04-18T14:14:36Z","logger":"barman-cloud-backup","msg":"2024-04-18 14:14:36,477 [1031] ERROR: Backup failed uploading data (An error occurred (InvalidArgument) when calling the CreateMultipartUpload operation: Unknown)","pipe":"stderr","logging_pod":"postgres-cluster-devops-playground-2"}

So my suspicion is that the backup isn't actually successful (which is a problem in it of itself, but one for CNPG to fix). Now I've stumbled into similar issues before with other services (Harbor docker registry for example) where "MultipartUpload" was failing, simply because my s3-compatible service does not support MultipartUpload ☹️ In Harbor we fixed it by setting a really high value for a field called "multipartcopythresholdsize", that way MultipartUpload is never initiated.

Is there a similar attribute for barman to allow basically disabling MultipartUpload? And more importantly, can it be configured with CNPG?

[alwinzwets]

Hi @Aransh I'm seeing the same errors with specifically Linode ObjectStorage. In addition to the CreateMultipartUpload error, I'm also getting this error on PutObject.

{"level":"info","ts":"2024-07-03T09:29:38Z","logger":"barman-cloud-backup","msg":"2024-07-03 09:29:38,723 [1513] ERROR: Backup failed uploading backup.info file (An error occurred (InvalidArgument) when calling the PutObject operation: Unknown)","pipe":"stderr","logging_pod":"database-cluster-2"}

Did you ever get barman working with Linode ObjectStorage?

[Aransh]

@alwinzwets Nope, had to use a custom solution instead...

[alwinzwets]

@Aransh fyi, for me it turned out the problem had nothing to do with multipart uploads but with the lack of SSE-S3 encryption support on Linode. See #957

[Aransh]

@Aransh fyi, for me it turned out the problem had nothing to do with multipart uploads but with the lack of SSE-S3 encryption support on Linode. See #957

I actually knew Linode object storage doesn't support encryption and this part won't work, had it disabled in all my tests. But as I noted in my comment, backups are indeed uploaded, but only partly as those that attempt to use multipart upload (which is automatic by file size) fail, leaving me with useless backups... You may want to check the size of the files you’re uploading, as they might stop working once they get big enough