generate_presigned_post failes when uploding large files

[alejoGT1202]

Describe the bug

I've implemented the generate_presigned_post for uploading files to my s3. I have tested it on my EKS Cluster with small video clips and it works as expected. However when I try to upload clips in the range of GB I get the following error:

HTTPSConnectionPool(host='my-bucket.s3.amazonaws.com', port=443): Max retries exceeded with url: /streamed (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2423)')))

I have tried setting verify=false on my request and setting my NO_PROXY env variable as they suggest here but so far none of this have helped me out to overcome the issue.

Expected Behavior

The POST request should upload the file to my bucket, if I'm complying with all the conditions that I set when the policy was created.

Current Behavior

When my script try to performs the POST request I get the following error:

HTTPSConnectionPool(host='my-bucket.s3.amazonaws.com',` port=443): Max retries exceeded with url: /streamed (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2423)')))

Reproduction Steps

Implement the presigned_post with the following conditions:

{'acl': 'private'},
['starts-with', '$Content-Type', 'video/'],
['content-length-range', 0, 50000000000], # 50Gb

Try to upload a large file to the returned policy using this sample code

import requests    # To install: pip install requests

# Generate a presigned S3 POST URL
object_name = 'OBJECT_NAME'
response = create_presigned_post('BUCKET_NAME', object_name)
if response is None:
    exit(1)

# Demonstrate how another Python program can use the presigned URL to upload a file
with open(object_name, 'rb') as f:
    files = {'file': (object_name, f)}
    http_response = requests.post(response['url'], data=response['fields'], files=files)
# If successful, returns HTTP status code 204
logging.info(f'File upload HTTP status code: {http_response.status_code}')

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.21.32

Environment details (OS name and version, etc.)

Python:3 Docker Image

[indrora]

This appears to be something involving the underlying SSL connection. Could you give us more on what environment you're running this in?

Can you generate a presigned URL and manually (e.g. using cURL) push content of a similar size to that?

[alejoGT1202]

I'm running it on my Amazon EKS cluster, some pods use the Python:3 Docker image and some other pods use the nvidia/cuda:11.6.0-base-ubuntu20.04. If I run the docker containers or the script locally in my computer, I overcome the error. The issue is when I try to run it on my cluster.

[tim-finnigan]

Linking related issue that might help: https://github.com/boto/boto3/issues/3477. It might also be worth updating as you're using an old version of boto3 (the latest version is 1.26.152 per the CHANGELOG). But if it works locally and not in your EKS cluster then the issue could be with the configuration of the cluster.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[indrora]

Yeah I can't tell if this is related to the hanging SSL connections that we have sometimes or what.

@alejoGT1202 Let me see if I fully understand: When you run your containers under a local Docker or some local K8s (minikube, etc.) it works fine, but when deployed under EKS, the presigned POST fails with an SSL error.

[alejoGT1202]

Yes, when I run my container under my local Docker it works as expected. But when I run the same container on my EKS cluster, I get the SSL error. The weird thing is that I get the SSL issue doing POST request to my S3, because I also make GET request using presigned urls and this doesn't give me any certificates error.

[lindi2]

I kept hitting the same problem and solved it by using 50 MB chunks with create_multipart_upload, generate_presigned_url and complete_multipart_upload.