upload_fileobj hangs indefinitely without returning error

[fitnycdigitalinitiatives]

Describe the bug

Hello,

We've recently run into a problem that seems a bit similar to #3657. Occasionally our application will stall out uploading objects to our bucket seemingly randomly using upload_fileobj without returning an error. We've run this application for years without experience this issue, but recent changes have included upgrading boto3 to 1.33.2 and RHEL to 9 (not sure if relevant to the issue). As suggested in the related issue, we turned on the debug logs and here's an example when the upload just completely hangs for hours on end:

DEBUG     2024-05-28 19:33:11  s3transfer.utils:utils:acquire:632:  Acquiring 0
DEBUG     2024-05-28 19:33:11  s3transfer.tasks:tasks:_wait_until_all_complete:205:  UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f94df942b20>}) about to wait for the following futures []
DEBUG     2024-05-28 19:33:11  s3transfer.tasks:tasks:_wait_until_all_complete:216:  UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f94df942b20>}) done waiting for dependent futures
DEBUG     2024-05-28 19:33:11  s3transfer.tasks:tasks:_execute_main:160:  Executing task UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f94df942b20>}) with kwargs {'client': <botocore.client.S3 object at 0x7f94e0189490>, 'config': <boto3.s3.transfer.TransferConfig object at 0x7f94df860a60>, 'osutil': <s3transfer.utils.OSUtils object at 0x7f94e02bcd90>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7f94e02bc3d0>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f94df942b20>}
DEBUG     2024-05-28 19:33:11  s3transfer.futures:futures:submit:318:  Submitting task CreateMultipartUploadTask(transfer_id=0, {'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f94e02bc3d0> for transfer request: 0.
DEBUG     2024-05-28 19:33:11  s3transfer.utils:utils:acquire:632:  Acquiring 0
DEBUG     2024-05-28 19:33:11  s3transfer.tasks:tasks:_wait_until_all_complete:205:  CreateMultipartUploadTask(transfer_id=0, {'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'extra_args': {}}) about to wait for the following futures []
DEBUG     2024-05-28 19:33:11  s3transfer.tasks:tasks:_wait_until_all_complete:216:  CreateMultipartUploadTask(transfer_id=0, {'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'extra_args': {}}) done waiting for dependent futures
DEBUG     2024-05-28 19:33:11  s3transfer.tasks:tasks:_execute_main:160:  Executing task CreateMultipartUploadTask(transfer_id=0, {'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7f94e0189490>, 'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'extra_args': {}}
DEBUG     2024-05-28 19:33:11  s3transfer.futures:futures:submit:318:  Submitting task UploadPartTask(transfer_id=0, {'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'part_number': 1, 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f94e02bc3d0> for transfer request: 0.
DEBUG     2024-05-28 19:33:11  s3transfer.utils:utils:acquire:632:  Acquiring 0
DEBUG     2024-05-28 19:33:11  urllib3.connectionpool:connectionpool:_make_request:474:  https://s3.us-east-1.amazonaws.com:443 "POST /BUCKET/packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z?uploads HTTP/1.1" 200 None
DEBUG     2024-05-28 19:33:11  s3transfer.utils:utils:release:645:  Releasing acquire 0/None
DEBUG     2024-05-28 19:33:11  s3transfer.tasks:tasks:_wait_until_all_complete:205:  UploadPartTask(transfer_id=0, {'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'part_number': 1, 'extra_args': {}}) about to wait for the following futures [<s3transfer.futures.ExecutorFuture object at 0x7f94e02d68b0>]
DEBUG     2024-05-28 19:33:11  s3transfer.tasks:tasks:_wait_until_all_complete:210:  UploadPartTask(transfer_id=0, {'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'part_number': 1, 'extra_args': {}}) about to wait for <s3transfer.futures.ExecutorFuture object at 0x7f94e02d68b0>
DEBUG     2024-05-28 19:33:11  s3transfer.tasks:tasks:_wait_until_all_complete:216:  UploadPartTask(transfer_id=0, {'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'part_number': 1, 'extra_args': {}}) done waiting for dependent futures
DEBUG     2024-05-28 19:33:11  s3transfer.tasks:tasks:_execute_main:160:  Executing task UploadPartTask(transfer_id=0, {'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'part_number': 1, 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7f94e0189490>, 'fileobj': <s3transfer.utils.ReadFileChunk object at 0x7f94df85bb50>, 'bucket': 'BUCKET', 'key': 'packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z', 'part_number': 1, 'extra_args': {}, 'upload_id': '2SeyzpFlJu4q6sWC9ma6r_.qparAJoyiPsNNcFW_Y_R3zbVs_G9bdPIVD_fLSy8gBhGYrawLwYUHZLunxLkDABbJL4DFTfhBY_Fj1RzR7prL6IHtDPO1UvpvBefVagKq'}
DEBUG     2024-05-28 19:33:11  urllib3.connectionpool:connectionpool:_make_request:474:  https://s3.us-east-1.amazonaws.com:443 "PUT /BUCKET/packages/d388/83aa/aa70/4879/944a/20b3/a601/3537/lb_ds_chf_000227-d38883aa-aa70-4879-944a-20b3a6013537.7z?uploadId=2SeyzpFlJu4q6sWC9ma6r_.qparAJoyiPsNNcFW_Y_R3zbVs_G9bdPIVD_fLSy8gBhGYrawLwYUHZLunxLkDABbJL4DFTfhBY_Fj1RzR7prL6IHtDPO1UvpvBefVagKq&partNumber=1 HTTP/1.1" 200 0

As you can see it just hangs after the initial PUT. In successful uploads after the first PUT, there's DEBUG 2024-05-28 19:33:05 s3transfer.utils:utils:release:645: Releasing acquire 0/None

There's nothing different about the files that it's failing on vs. the files it uploads without issue. The vast majority of uploads are successful. Just these occasional uploads stall without error.

Expected Behavior

Uploads should fail with an error.

Current Behavior

See above.

Reproduction Steps

Not able to reproduce, happens randomly when using upload_fileobj. The application uses this method:

boto_args = {
    "service_name": "s3",
    "region_name": "REGION",
    "config": config,
}
resource = boto3.resource(**boto_args)
bucket = resource.Bucket("BUCKET")
with open('filename', 'rb') as data:
    bucket.upload_fileobj(data, 'mykey')

Possible Solution

No response

Additional Information/Context

No response

SDK version used

boto3 1.33.2

Environment details (OS name and version, etc.)

rhel 9