caused by: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive

[anacrolix]

I suspect my DB is too large (5x1000=5GB?), my DB is 36 GB, running replicate I get:

litestream initialization complete
some-file(s3): monitor error: MultipartUpload: upload multipart failed
        upload id: x
caused by: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
        status code: 400, request id: x host id: x

[benbjohnson]

Hmm, that's a new one. Yeah, it looks like your DB is too big. :)

Looks like the PartSize and MaxUploadParts on the s3manager.Uploader needs to be set within the S3 client. However, the SDK docs say the default MaxUploadParts is 10,000 so I'm not sure why it's showing 1000 in the error.

It'd be nice to set this automatically although Litestream runs the snapshot through an LZ4 compressor so we don't know the final size. We could probably estimate it. Or it could just be a config option since most folks probably aren't running 36GB databases.

[danp]

@anacrolix Just double checking: you are connecting to S3 proper and not some other S3-compat service? Searching for Part number must be an integer between 1 and 1000, I mostly see mentions of Scaleway.

[anacrolix]

@danp thank you for picking this. Yes I am using Scaleway! @benbjohnson how can I modify it so that the parts are bigger? I guess my max parts are 1000 for scaleway? Is there somewhere that is set so it can adjust the part size to suit?

[benbjohnson]

@anacrolix Litestream doesn't currently support setting the part size. It shouldn't be an especially difficult change though:

1. Add field to ReplicaConfig
2. Pass it to the ReplicaClient in newS3ReplicaFromConfig()
3. Set it on the s3manager.Uploader inside s3.ReplicaClient.Init()

It may be useful to have a different part size for snapshots versus WAL pages but maybe that's overthinking it.

[anacrolix]

It looks like the S3 uploader will try to set part size for you if it can seek the input reader. However I think the input is passed through a pipe for LZ4, so seeking isn't possible. Based on this, setting max upload parts to 1000, and adjusting part size to fit seems like the best way forward.