localstack
bug: Firehose delivery stream HTTP destination cannot use self-signed certificates for HTTPS
Is there an existing issue for this?
- [X] I have searched the existing issues
Current Behavior
When trying to use http://, Firehose complains that the HTTP destination URL must start with https://
When using https:// with a self-signed certificate, such as with the HTTPS echo server Docker image mendhak/http-https-echo and the URL https://host.docker.internal:8443/, there is an error which does not allow the server to be used:
urllib3.exceptions.MaxRetryError: MyHTTPSConnectionPool(host='host.docker.internal', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/firehose/provider.py", line 695, in _put_records
requests.post(url, json=record_to_send, headers=headers)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/adapters.py", line 698, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: MyHTTPSConnectionPool(host='host.docker.internal', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))
2025-01-14T04:10:34.266 ERROR --- [et.reactor-1] l.aws.handlers.logging : exception during call chain: MyHTTPSConnectionPool(host='host.docker.internal', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))
2025-01-14T04:10:34.267 ERROR --- [et.reactor-2] l.aws.handlers.logging : exception during call chain: An error occurred (InternalError) when calling the PutRecord operation (reached max retries: 4): exception while calling firehose.PutRecord: MyHTTPSConnectionPool(host='host.docker.internal', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))
2025-01-14T04:10:34.267 INFO --- [et.reactor-2] localstack.request.aws : AWS logs.PutLogEvents => 500 (InternalError)
Expected Behavior
It should be possible to temporarily disable HTTPS certificate verification for Firehose requests when using the HTTP destination.
How are you starting LocalStack?
With the localstack script
Steps To Reproduce
How are you starting localstack
localstack start
Client commands
aws --endpoint-url=http://localhost:4566 logs put-log-events \
--log-group-name /aws/lambda/test-function \
--log-stream-name example-log-stream \
--log-events timestamp=$(date +%s000),message="Test log message"
Environment
- OS: MacOS 14.6.1
- LocalStack:
LocalStack version: 4.0.4.dev99
LocalStack Docker image sha: f55192023ba338ed9a8f0a0eae604ef88e69602b4803fec623c028a4ff041c17
LocalStack build date: 2025-01-10
LocalStack build git hash: f95f29208
Anything else?
The workaround I used was to add verify=False to the requests.post call at line 695 of firehose/provider.py and then restart the localstack container.
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.
Hello 👋! It looks like this issue hasn’t been active in longer than five months. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.