When aws s3 cp --recursive fail to download file, the failure message is hidden in the middle of the logs

[usamec]

trafficstars

Describe the bug

When I run: aws s3 cp --recursive s3://something/ local_path/

And some file fails to download (when internet is shaky), the reported error is hidden in the middle of the logs: download failed: s3://something/file Could not connect to the endpoint URL: "https://something.s3.eu-west-1.amazonaws.com/file"

Expected Behavior

There is summary of errors at the end of log.

Current Behavior

See description. Error is hidden in the middle of logs and less files are downloaded.

Reproduction Steps

See description. Get shaky internet for it.

Possible Solution

See expected behaviour. Add summary of error at the end.

Additional Information/Context

No response

CLI version used

aws-cli/2.2.5 Python/3.8.8 Linux/4.15.0-176-generic exe/x86_64.ubuntu.18 prompt/off

Environment details (OS name and version, etc.)

Ubuntu

[tim-finnigan]

Hi @usamec thanks for reaching out. Can you provide your full debug logs by adding --debug to the command? (And with any sensitive information redacted.). That could help give us more insight into the issue.

If you’re experiencing network connectivity issues then that would explain why the service is interrupted. You could try configuring a retry mode as documented here.

[usamec]

Hi @tim-finnigan, I think debug logs would be too long and useless. Also we are aware of network connectivity issues, that is a normal thing.

The problem is simple: Logs from cp command contain a lot of lines like:

download: s3://bucket/fileX to fileX
download: s3://bucket/fileY to fileY

End of logs looks like:

download: s3://bucket/fileZ to fileZ

And if somewhere in the middle of logs (e.g. you are downloading 10000 files), there is a failure, you won't notice.

All I am asking for is to output summary of form:

Summary: 
Downloaded 12345 files. 
There were 5 failures: s3://bucket/fileABC, s3://bucket/fileABD, s3://bucket/fileABE, s3://bucket/fileABF, s3://bucket/fileABG

[tim-finnigan]

Thanks @usamec for clarifying that, I'm going to change the bug label to feature request then. We ask others in the community to 👍 your issue if this is something they also want to see.

[kdaily]

Hi @usamec,

Thanks for the feature request. I understand it doesn't help you find out which file was the issue (or how many), but you can determine if something failed. In your situation, the return code of the operation will be 1. This is documented here:

https://docs.aws.amazon.com/cli/latest/topic/return-codes.html#cli-aws-help-return-codes

1 -- Limited to s3 commands, at least one or more s3 transfers failed for the command executed.

Hope that helps!

[orphefs]

Hello, I am running awscli version 1.18.69-1ubuntu0.20.04.1, and when trying a command

aws s3 cp s3://bucket/path/to/data/ local/folder/

and path/to/data does not exist, I obtain the following error:

fatal error: An error occurred (404) when calling the HeadObject operation: Key "path/to/data" does not exist

with exit status 1.

However, if I run the same command with the flag --recursive, as in

aws s3 cp s3://bucket/path/to/data/ local/folder/ --recursive

I don't receive an error, and exit status is 0.

I need to terminate the parent process if cp fails, so having the CLI return the correct error code upon failure would solve my problem. Thank you!

[frivas-at-navteca]

Hello, I am running awscli version 1.18.69-1ubuntu0.20.04.1, and when trying a command

aws s3 cp s3://bucket/path/to/data/ local/folder/

and path/to/data does not exist, I obtain the following error:

fatal error: An error occurred (404) when calling the HeadObject operation: Key "path/to/data" does not exist

with exit status 1.

However, if I run the same command with the flag --recursive, as in

aws s3 cp s3://bucket/path/to/data/ local/folder/ --recursive

I don't receive an error, and exit status is 0.

I need to terminate the parent process if cp fails, so having the CLI return the correct error code upon failure would solve my problem. Thank you!

I am having exactly the same issue. In addition to this:

try:
    cmd_split = shlex.split(f"aws2 s3 cp s3://bucket/non_existing_folder/ download --profile xxx")
    process = subprocess.Popen(cmd_split, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
except client.exceptions.NoSuchBucket:
    print(f"The specified Bucket: {bucket} has not been found")
    message = f"The specified Bucket: {bucket} has not been found"
except client.exceptions.NoSuchKey:
    print(f"The specified Key: {key} has not been found")
    message = f"The specified Key: {key} has not been found"
except exceptions.ClientError as error:
    print(f"There has been an error with the S3 client => {error}")
    message = f"There has been an error with the S3 client => {error}"
except subprocess.CalledProcessError as exc:
    print(f"Program failed {exc.returncode} - {exc}")
    message = f"Program failed {exc.returncode} - {exc}"
except subprocess.TimeoutExpired as exc:
    print(f"Program timed out {exc}")
    message = f"Program timed out {exc}"
except NotEnoughSpaceOnDevice as exc:
    print(f"{exc}")
    message = f"{exc}"
except ApplicationNotFound as exc:
    print(f"{exc}")
    message = f"{exc}"
except Exception as exc:
    print(f"Exception {exc}")
    message = f"Exception {exc}"
else:
    success = True

I would expect when doing:

out, error = process.communicate()
print(out)
print(error)

To show something, but both are empty. Any hints on why this is happening like that?

Thank you very much! Have a great day!