aptly
aptly copied to clipboard
aptly-dev
413 Entity Too Large Error when publishing Debian Bullseye mirror to S3 (Ceph Object Storage) with large file sizes
I have a Debian Bullseye mirror that I want to publish to an S3-compatible storage (Ceph). However, I'm encountering an issue when trying to publish the mirror. The S3-compatible storage has a maximum size limit for a single part of 5GB and for multipart uploads, it's 500MB, with a minimum size for multipart uploads being 5MB. When I attempt to publish the mirror, I receive a "413 Request Entity Too Large" error.
Here's the error log:
3 / 119009 [>-------------------------------------------------------------------] 0.00% 11h24m30s2023/05/10 09:40:17 DEBUG: Request s3/PutObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /apt-mirror/debian/bullseye-test/pool/main/0/0ad-data/0ad-data_0.0.23.1-1.1_all.deb HTTP/1.1
Host: s3.ir-thr-at1.arvanstorage.ir
User-Agent: aws-sdk-go/1.25.0 (go1.17.11; linux; amd64)
Content-Length: 701833824
Authorization: AWS4-HMAC-SHA256 Credential=7cd89524-2748-4483-834a-d5dc48d85aaf/20230510/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-md5;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-meta-md5, Signature=xxxx
Content-Md5: srblUQiYq/Du552kiZX5Lw==
Expect: 100-Continue
X-Amz-Acl: private
X-Amz-Content-Sha256: afb3f0ddaceb36dc2d716d83d7fee4ada419511a948e4a06fa44bbc1b486e2c0
X-Amz-Date: 20230510T094009Z
X-Amz-Meta-Md5: b2b6e5510898abf0eee79da48995f92f
Accept-Encoding: gzip
-----------------------------------------------------
2023/05/10 09:40:17 DEBUG: Response s3/PutObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 413 Request Entity Too Large
Content-Length: 189
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization,Accept,Origin,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range
Access-Control-Allow-Methods: GET,POST,OPTIONS,PUT,DELETE,PATCH
Access-Control-Allow-Origin: *
Content-Type: text/html
Date: Wed, 10 May 2023 09:40:17 GMT
Server: ArvanCloud
Server-Timing: total;dur=50
X-Request-Id: 11011f780026e6a8710dbaf03756598d
X-Sid: 2063
-----------------------------------------------------
ERROR: unable to publish: unable to process packages: error uploading af/b3/f0ddaceb36dc2d716d83d7fee4ad_0ad-data_0.0.23.1-1.1_all.deb to S3: us-east-1:apt-mirror/: debian/bullseye-test/pool/main/0/0ad-data/0ad-data_0.0.23.1-1.1_all.deb: SerializationError: failed to unmarshal error message
status code: 413, request id: , host id:
caused by: UnmarshalError: failed to unmarshal error message
00000000 3c 68 74 6d 6c 3e 0d 0a 3c 68 65 61 64 3e 3c 74 |<html>..<head><t|
00000010 69 74 6c 65 3e 34 31 33 20 52 65 71 75 65 73 74 |itle>413 Request|
00000020 20 45 6e 74 69 74 79 20 54 6f 6f 20 4c 61 72 67 | Entity Too Larg|
00000030 65 3c 2f 74 69 74 6c 65 3e 3c 2f 68 65 61 64 3e |e</title></head>|
00000040 0d 0a 3c 62 6f 64 79 3e 0d 0a 3c 63 65 6e 74 65 |..<body>..<cente|
00000050 72 3e 3c 68 31 3e 34 31 33 20 52 65 71 75 65 73 |r><h1>413 Reques|
00000060 74 20 45 6e 74 69 74 79 20 54 6f 6f 20 4c 61 72 |t Entity Too Lar|
00000070 67 65 3c 2f 68 31 3e 3c 2f 63 65 6e 74 65 72 3e |ge</h1></center>|
00000080 0d 0a 3c 68 72 3e 3c 63 65 6e 74 65 72 3e 6f 70 |..<hr><center>op|
00000090 65 6e 72 65 73 74 79 2f 31 2e 31 39 2e 39 2e 31 |enresty/1.19.9.1|
000000a0 3c 2f 63 65 6e 74 65 72 3e 0d 0a 3c 2f 62 6f 64 |</center>..</bod|
000000b0 79 3e 0d 0a 3c 2f 68 74 6d 6c 3e 0d 0a |y>..</html>..|
caused by: expected element type <Error> but have <html>
The object I've get this error on is:
ls -lah af/b3/f0ddaceb36dc2d716d83d7fee4ad_0ad-data_0.0.23.1-1.1_all.deb
-rw-r--r-- 1 debian debian 670M Dec 16 11:45 af/b3/f0ddaceb36dc2d716d83d7fee4ad_0ad-data_0.0.23.1-1.1_all.deb
It seems that the error is caused by the file being too large to be uploaded. Is there a way to configure Aptly to handle this situation, or is this a bug that needs to be addressed? Any guidance or assistance would be greatly appreciated.
