s3 head_object returns ClientError 404 instead of NoSuchKey error

[carsonip]

What issue did you see ?

When the key does not exist in s3, s3 head_object returns a ClientError 404 instead of NoSuchKey error, making it harder to catch. Also, the doc says it should throw a NoSuchKey error. But get_object returns NoSuchKey.

Steps to reproduce

client.head_object(
    Bucket=bucket,
    Key=key,
)  # throws ClientError with code 404

client.get_object(
    Bucket=bucket,
    Key=key,
)  # throws NoSuchKey

Is this expected? If so, should the docs be modified? Thanks.

[swetashre]

@carsonip - Thank you for your post. I am able to reproduce the behavior. The issue here is that this is a HEAD request, which does not contain a response body. In these scenarios we just use the status code message as the error message.

As boto3 documentation is generated from API Reference i will reach out to the service team about the issue. I will post here when i get any update.

[pickfire]

Looks like this issue happened from way back around 1.9.201.

[dcworldwide]

Might be related, by I get intermittent NoSuchKey despite the file absolutely existing in s3. Eventual consistency isn't the issue, as the file was uploaded hours prior.

[f0o]

@swetashre any update on this? It's been a year now and we're still experiencing this issue.

[adonig]

As a temporary workaround (implying the bug will get fixed one day) you can catch the ClientError like this:

import botocore
import boto3

s3_client = boto3.client("s3")
try:
    response = s3_client.head_object(Bucket="mybucket", Key="mykey")
except s3_client.exceptions.NoSuchKey:  # type: ignore
    pass   # TODO: Handle the exception here.
except botocore.exceptions.ClientError as err:
    # NOTE: This case is required because of https://github.com/boto/boto3/issues/2442
    if err.response["Error"]["Code"] == "404":
        pass  # TODO: Handle the exception here again.

That way your code will still work as expected in case the bug gets fixed.

[ryanbagwell]

We ran into this recently. We expected the call to return a value if no object was found like the docs say, but instead encountered the ClientError exception which resulted in a 500. Catching the error is a fine workaround but the docs should be corrected as others have stated.

[tim-finnigan]

Checking in - I believe the docs have been updated since this original issue was opened. In the head_object and get_object boto3 docs it now says:

• If you have the s3:ListBucket permission on the bucket, Amazon S3 returns an HTTP status code 404 ("no such key") error.
• If you don’t have the s3:ListBucket permission, Amazon S3 returns an HTTP status code 403 ("access denied") error.

If that still doesn't clarify the behavior then any further requests for improvements to the documentation should be redirected to the S3 team because they own their API documentation as mentioned here earlier.

There is a Provide feedback link at the bottom of every API page where you can send direct feedback to the appropriate team. Alternatively you can reach out through AWS Support or we can forward requests internally. Please let us know if you had any follow up questions on this.

[Erouan50]

@tim-finnigan you are right, and on the S3 head_object is even more explicit:

A HEAD request has the same options as a GET action on an object. The response is identical to the GET response except that there is no response body. Because of this, if the HEAD request generates an error, it returns a generic 404 Not Found or 403 Forbidden code. It is not possible to retrieve the exact exception beyond these error codes.

However, the boto3 still references the S3.Client.exceptions.NoSuchKey in the list of exception. Shouldn't be removed to avoid any confusion?

[zzJinux]

@tim-finnigan https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html#API_HeadObject_Errors also references NoSuchKey. I have sent a feedback from the url.