Maximum number of parts per upload constraint (S3)

[avoidik]

Hello,

I would like to understand if there's a safety logic to prevent more than 10_000 (ref. link) multi-part chunks being uploaded to AWS S3 bucket. It seems I'm facing with an edge-case scenario where I have more than 10_000 chunks, as a result I have got the following error:

:body              => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidArgument</Code><Message>Part number must be an integer between 1 and 10000, inclusive</Message><ArgumentName>partNumber</ArgumentName><ArgumentValue>10001</ArgumentValue><RequestId>removed</RequestId><HostId>removed</HostId></Error>"
:cookies           => [
]
:headers           => {
  "Connection"       => "close"
  "Content-Type"     => "application/xml"
  "Date"             => "Tue, 06 May 2025 04:14:43 GMT"
  "Server"           => "AmazonS3"
  "x-amz-id-2"       => "removed"
  "x-amz-request-id" => "removed"
}
:host              => "removed.s3.eu-central-1.amazonaws.com"
:local_address     => "x.x.x.x"
:local_port        => 49484
:method            => "PUT"
:omit_default_port => false
:path              => "/huge_file.tar"
:port              => 443
:query             => {
  "partNumber" => 10001
  "uploadId"   => "removed"
}
:reason_phrase     => "Bad Request"
:remote_ip         => "y.y.y.y"
:scheme            => "https"
:status            => 400
:status_line       => "HTTP/1.1 400 Bad Request\r\n"

I completely understand that it's possible to adhere to 10_000 limit by adjusting the multipart_chunk_size property. However, it is not always feasible to know how much data is going to be uploaded upfront.

Cheers

[geemus]

@avoidik Hey, that error is just being passed through from AWS, so I'm afraid I don't know the specific reason any more than you do at this point. If you'd like to know more I'd suggest reaching out to them directly to see if they can better explain. Sorry that I don't have more info for you.

[avoidik]

Thanks for coming back with the response @geemus! I understand that's an AWS S3 backend error, thank you. The original question was about the library itself, it seems there's no logic neither to handle >10000 chunks exception nor condition to ensure that object size / chunks size < 10001. Is this left up to a library user to implement?

[geemus]

@avoidik Ah, got it. Thanks for clarifying that, I definitely didn't understand the request originally. It does seem like the experience of using it could be better, at least in cases where we know up front that object size / chunk size > 10000. It seems like it should probably just raise an error in that case I guess? Is that what you had in mind?

[avoidik]

@geemus I was trying to understand this exception from the developer's experience point of view, specifically what developers can anticipate from this library. You have implemented the validation of minimum (5MB) and maximum (5TB) chunk size. Facing this kind of exceptions makes me naturally wonder, have you implemented validation checks for other constraints or not. Do you expect this validation to happen on a consumer's side or it's something overlooked on the library side? If former, I'll close the issue (implying that chunks maximum limit should be calculated by myself), and if latter - then it's a minor addition to the list of validations implemented in this library.

[geemus]

@avoidik sure. I think it probably makes sense to add this validation to the library.