cli
cli copied to clipboard
Certain requests run into timeout
Describe the bug
When hosting min.io (self-hosted S3) via loophole, I can upload large files (multiple 100M or multiple GB) via s3cmd without any issues, but when uploading them via aws-cli, I get timeouts and the upload fails.
To Reproduce Steps to reproduce the behavior:
- Host
min.io
e.g. via docker:docker run -v "$HOME/.minio:/data -p 9000:9000 -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" minio/minio server /data
- Start site via
loophole 9000
- Create S3 bucket via aws-cli:
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY aws s3 mb --endpoint-url https://${LOOPHOLE_SITE}.loophole.site s3://test
- Check successful upload via s3cmd:
s3cmd --access_key=AKIAIOSFODNN7EXAMPLE --secret_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --continue-put --host ${LOOPHOLE_SITE}.loophole.site --host-bucket '${LOOPHOLE_SITE}.loophole.site' put SomeLargeFile.zip s3://test/SomeLargeFile.zip
- Run into errors when uploading via aws cli:
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY aws s3 cp --endpoint-url https://${LOOPHOLE_SITE}.loophole.site/ SomeLargeFile.zip s3://test/SomeOtherLargeFile.zip
First, download speed goes down from ~1.4M/sec to ~500K/sec and progress hangs every now and then. After some time of no progress, it fails with this message:
upload failed: SomeLargeFile.zip to s3://test/SomeOtherLargeFile.zip Connection was closed before we received a valid response from endpoint URL: "https://${LOOPHOLE_SITE}.loophole.site/max/SomeOtherLargeFile.zip?uploadId=2992f2aa-d82b-4874-bc74-3064a5ae79c5&partNumber=2".
Expected behavior
Upload via s3cmd
and aws-cli
both work.
Screenshots N/A
Environment
- OS: macOS
- Version:
loophole version 1.0.0-beta.7 (95033a34197863fe1074bf148758b30d16477fb6)
Additional context The only difference I can see on a low level is that s3cmd is uploading a single part at a time only and aws-cli is uploading multiple parts in parallel. Maybe this helps. Also somehow suspect the handling of connection close to be the issue.