int size errors when uploading large file (8GB) on 32-bit machine (raspberry pi)

[fireduck64]

Error:

[Exception] Python int too large to convert to C long [Thread Failure] Python int too large to convert to C long

Command line: s4cmd put -c 4 /mnt/xfer/largefile s3://bucketarray/largefile

Platform: Debian on armv7l (Raspbian) Software:

s4cmd=2.1.0+ds-1

Hypothesis: Some code somewhere is attempting to convert a large python into into a C 'long' which can vary in size between platforms. Probably needs to be a long long or something.

[klou]

The problem is here: https://github.com/bloomreach/s4cmd/blob/e74e0e7cc666d39af054d231d0a84f817dbab2fa/s4cmd.py#L1296-L1307

In particular:

https://github.com/bloomreach/s4cmd/blob/e74e0e7cc666d39af054d231d0a84f817dbab2fa/s4cmd.py#L1304

When the chunk is larger than a 32-bit system can handle, then it crashes. This is probably the case for all files over 2 GB.

One workaround seems to be setting --max-singlepart-upload-size to something smaller than 2 GB (and it defaults to 4.5 gb).