Boto3 s3 Client head_object Warning Failed to parse headers [NoBoundaryInMultipartDefect()],

[Jun711]

Describe the bug I am using boto3 to do head_object request in AWS Lambda Python 3.9 Runtime. The head_object request seems to be ok as I get 200 response code but there is this warning message that prints after each request that increases my CloudWatch usage. Failed to parse headers (url=MY_S3_FILE_URL): [NoBoundaryInMultipartDefect()], unparsed data: ''

This closed issue seems related If updating Boto3 would fix it, how do I make Lambda use that version of Boto3?

s3Client = boto3.client('s3')
def checkIfKeyExists(s3Bucket, filePath):
    try:
        res = s3Client.head_object(Bucket=s3Bucket, Key=filePath)
        return True
    except Exception as e:
        print('e', e)
        return False

This is the response with values removed

{'ResponseMetadata': {'RequestId': '', 'HostId': '', 'HTTPStatusCode': 200, 
'HTTPHeaders': {'x-amz-id-2': '', 'x-amz-request-id': '', 'date': 'Sat, 05 Feb 2022 02:20:39 GMT', 
'last-modified': 'Sat, 05 Feb 2022 02:20:38 GMT', 'x-amz-expiration': 'expiry-date="Mon, 07 Feb 2022 00:00:00 GMT", 
rule-id="Expire rule"', 'etag': '""', 'accept-ranges': 'bytes', 'content-type': 'multipart/form-data', 'server': 'AmazonS3', 
'content-length': '20922'}, 'RetryAttempts': 0}, 'AcceptRanges': 'bytes', 
'Expiration': 'expiry-date="Mon, 07 Feb 2022 00:00:00 GMT", rule-id="Expire rule"',
 'LastModified': datetime.datetime(2022, 2, 5, 2, 20, 38, tzinfo=tzutc()), 'ContentLength': 20922, 'ETag': '""', 
 'ContentType': 'multipart/form-data', 'Metadata': {}}

Steps to reproduce AWS Lambda Python3.9 runtime, not sure about boto3 version as it uses the preinstalled one so I think it should be the last stable version?

s3Client = boto3.client('s3')
def checkIfKeyExists(s3Bucket, filePath):
    try:
        res = s3Client.head_object(Bucket=s3Bucket, Key=filePath)
        return True
    except Exception as e:
        print('e', e)

Unexpected behavior I didn't expect it to print warning after each request

Debug logs

[WARNING] 2022-02-05T02:23:51.759Z b0cea05b-010b-490d-9c92-2d2252342b52 Failed to parse headers (url=MY_S3_FILE_URL): [NoBoundaryInMultipartDefect()], unparsed data: ''
Traceback (most recent call last):
File "/var/runtime/urllib3/connectionpool.py", line 465, in _make_request
assert_header_parsing(httplib_response.msg)
File "/var/runtime/urllib3/util/response.py", line 91, in assert_header_parsing
raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)
urllib3.exceptions.HeaderParsingError: [NoBoundaryInMultipartDefect()], unparsed data: ''

[kdaily]

Hi @Jun711,

Thanks for writing. According to this, the Python version on Lambda is boto3-1.18.55 botocore-1.21.55, which is well beyond the previously reported error.

https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html

You can confirm by adding boto3.set_stream_logger() to your code (reference). Be aware that it logs at the wire level, so be sure to not enable in a production system that may contain sensitive information.

Looking into it further.

[kdaily]

@Jun711,

Can you also confirm what version of urllib3 you are using?

[Jun711]

@kdaily I added logging and I could see that it is using boto3-1.18.55 botocore-1.21.55

'User-Agent': 'Boto3/1.18.55 Python/3.9.8 Linux/5.10.76-linuxkit exec-env/AWS_Lambda_python3.9 Botocore/1.21.55'}

I didn't install urllib3 so it is using the one installed by boto3. urllib3 version is 1.26.6

[kdaily]

Hi @Jun711,

Thanks for that information. I think I still need to see the output of the debugging statements, specifically to get a request ID. I may need to reach out to the S3 team. I'm not sure how you're getting a 'content-type': 'multipart/form-data' back from a head object request.

[Jun711]

@kdaily yea, sure. I removed signature and some info. If there is some sensitive info that needs to be deleted, please let me know. Thank you s3head-request-debug-log.txt

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[Jun711]

@kdaily sorry, it seems to be still happening. When I ran locally using sam, I didn't notice that but I found it in CloudWatch log

[tim-finnigan]

Checking in as there hasn't been any update in a while. @Jun711 is this still an issue you are seeing? Do you have any updated logs or code snippets you could share that might provide more insight into this?

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[ferruzzi]

Hi @tim-finnigan and @kdaily. We have run into this in Airflow as well, it's still an issue. Tasks are completing as expected, but the error message is getting printed to the logs anyway and causing some confusion. There are more details in the issue that Taragolis linked last week. Let me know what you need from me.