botocore.exceptions.SSLError: SSL validation failed on aws3 endpoint

[snirbenyosef]

Describe the bug i'm trying to download from s3 public bucket and got ssl error, while 2 weeks ago everything works and i didnt change anything on my workspace. i'm not sure why?

Steps to reproduce

    import boto3
    import botocore
    import urllib3

    from botocore.handlers import disable_signing

    s3_resource = boto3.resource(
                "s3",
                endpoint_url="https://s3.amazonaws.com",
                verify=False,
                region_name=None,
                aws_access_key_id=None,
                aws_secret_access_key=None,
            )
    s3_resource.meta.client.meta.events.register('choose-signer.s3.*', disable_signing)
    
    s3_resource.Bucket("mybucket").download_file(
                    "logo_detection_data/images/1.jpg", "/logo_detection_data/images/1.jpg", ExtraArgs=None)

Expected behavior download from s3 amazon public bucket.

Debug logs botocore.exceptions.SSLError: SSL validation failed for https://s3.amazonaws.com/libhub-readme/logo_detection_data/images/1.jpg [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

[kdaily]

Hi @snirbenyosef,

Sorry to hear you're having an issue. Can you please post debug logs using boto3.set_stream_logger(''), redacting any sensitive information?

Does the network you're on use a certificate proxy of any kind?

Also, it seems that you're disabling signing to download from the public bucket. The preferred way to do this is by configuring the session to use the UNSIGNED signature version:

import boto3
import botocore
from botocore import UNSIGNED
from botocore.config import Config

s3_resource = boto3.resource("s3", config=Config(signature_version=UNSIGNED))

[snirbenyosef]

@kdaily

Does the network you're on use a certificate proxy of any kind? i'm not sure but i didnt change anything in my workspace last 2 weeks.

2022-03-08 07:20:47,012 root [DEBUG] CMD line args: {'debug': False, 'endpoint': 'https://s3.amazonaws.com', 'region_name': None, 'command': 'download', 'bucket': 'mybucket', 'bucketname': True, 'localdir': '/cnvrg', 'overwrite': False, 'versionid': None, 'filename': None, 'prefix': 'logo_detection_data/', 'cnvrg_dataset': None, 'func': <function cmd_download at 0x7f68a9acbbf8>}
https://s3.amazonaws.com
None
None
None
2022-03-08 07:20:47,069 root [DEBUG] Checking if bucket exist: mybucket
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 371, in connect
    ssl_context=context,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/botocore/httpsession.py", line 394, in send
    chunked=self._chunked(request.headers),
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 386, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 371, in connect
    ssl_context=context,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "s3-connector.py", line 922, in <module>
    main()
  File "s3-connector.py", line 915, in main
    args.func(s3, args)
  File "s3-connector.py", line 842, in cmd_download
    if not s3.check_bucket_exist(args.bucket):
  File "s3-connector.py", line 414, in check_bucket_exist
    self.s3_resource.meta.client.head_bucket(Bucket=bucket_name)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 706, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 725, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.6/dist-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.6/dist-packages/botocore/endpoint.py", line 137, in _send_request
    success_response, exception):
  File "/usr/local/lib/python3.6/dist-packages/botocore/endpoint.py", line 255, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/usr/local/lib/python3.6/dist-packages/botocore/hooks.py", line 357, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/lib/python3.6/dist-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/usr/local/lib/python3.6/dist-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python3.6/dist-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/usr/local/lib/python3.6/dist-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/usr/local/lib/python3.6/dist-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/lib/python3.6/dist-packages/botocore/endpoint.py", line 199, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/lib/python3.6/dist-packages/botocore/endpoint.py", line 268, in _send
    return self.http_session.send(request)
  File "/usr/local/lib/python3.6/dist-packages/botocore/httpsession.py", line 412, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://s3.amazonaws.com/mybucket [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

[kdaily]

Hi @snirbenyosef,

Thanks for the logs. Looks like they're truncated though, there should be a lot more information about what boto3/botocore are doing. Are you sure you added boto3.set_stream_logger('') to your code?

Can you also provide details on the version of boto3/botocore you're using?

You would not have made any changes regarding certificate proxies or networking, but your system administrator might have. Can you replicate the error on another workspace in the same network environment?

Another thing you can check is to force which certificate bundle to use by setting the AWS_CA_BUNDLE environment variable or ca_bundle configuration file parameter. You would provide the path to your certificates. A certificate bundle is provided and used by default with botocore.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[dmitrytretyakov]

@snirbenyosef , did you find the reason? I faced the same issue using ansible community aws collection and investigating src code led me to botocore

[dmitrytretyakov]

the reason in my case and probably in @snirbenyosef is botocore of 1.23.0 is not compatible with python 3.6 (at least with verify=False option) Is that expected @kdaily ?

[nateprewitt]

Hi @dmitrytretyakov, could you clarify what isn’t compatible with Python 3.6? We’ll need logs as @kdaily requested above to verify the issue.

[dmitrytretyakov]

In case of using invalid certificates and trying to skip validation:

import boto3

client = boto3.client('s3', 
    aws_access_key_id='ID',
    aws_secret_access_key='reallysecret',
    endpoint_url='https://myaws',
    verify=False
)
client.list_buckets()

1. Python 3.6.9 boto3 (1.20.1) botocore (1.22.12) IS OK

2. Python 3.6.9 boto3 (1.20.1) botocore (1.23.1) IS NOT OK:

Traceback. Click to expand

Traceback (most recent call last): File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn conn.connect() File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/connection.py", line 371, in connect ssl_context=context, File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket return context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/usr/lib/python3.6/ssl.py", line 817, in __init__ self.do_handshake() File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake self._sslobj.do_handshake() File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/httpsession.py", line 394, in send chunked=self._chunked(request.headers), File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 386, in increment raise six.reraise(type(error), error, _stacktrace) File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise raise value.with_traceback(tb) File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in validate_conn conn.connect() File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/connection.py", line 371, in connect ssl_context=context, File "/opt/k8s-module-venv/lib/python3.6/site-packages/urllib3/util/ssl.py", line 386, in ssl_wrap_socket return context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/usr/lib/python3.6/ssl.py", line 817, in init self.do_handshake() File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake self._sslobj.do_handshake() File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "boto3-bug.py", line 8, in print(client.list_buckets()) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/client.py", line 391, in _api_call return self._make_api_call(operation_name, kwargs) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/client.py", line 706, in _make_api_call operation_model, request_dict, request_context) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/client.py", line 725, in _make_request return self._endpoint.make_request(operation_model, request_dict) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request return self._send_request(request_dict, operation_model) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/endpoint.py", line 137, in _send_request success_response, exception): File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/endpoint.py", line 255, in _needs_retry caught_exception=caught_exception, request_dict=request_dict) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/hooks.py", line 357, in emit return self._emitter.emit(aliased_event_name, **kwargs) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit return self._emit(event_name, kwargs) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit response = handler(**kwargs) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in call if self._checker(attempts, response, caught_exception): File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in call caught_exception) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 277, in _should_retry return self._checker(attempt_number, response, caught_exception) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in call caught_exception) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in call attempt_number, caught_exception) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception raise caught_exception File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/endpoint.py", line 199, in _do_get_response http_response = self._send(request) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/endpoint.py", line 268, in _send return self.http_session.send(request) File "/opt/k8s-module-venv/lib/python3.6/site-packages/botocore/httpsession.py", line 412, in send raise SSLError(endpoint_url=request.url, error=e) botocore.exceptions.SSLError: SSL validation failed for https://myaws/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

[tim-finnigan]

Checking in - support for Python 3.6 has since been deprecated as of May 2022 per this announcement. If this is still an issue in Python 3.7+ please let us know.

[diptripa]

@tim-finnigan, not sure if its the same but getting this error woth below versions: boto3~=1.28.10 botocore~=1.31.10 python~= 3.11 openssl~=3.1.1 ` urllib3.exceptions.SSLError: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1002)

During handling of the above exception, another exception occurred: botocore.exceptions.SSLError: SSL validation failed for https://alln-cloud-storage-1.cisco.com/ [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1002) `

[zackees]

Hitting this constantly with Python 3.11.7.

Do I really have to use verify=False on the official boto library?