vector
vector copied to clipboard
aws_kinesis_firehose source with tls enabled generate errors on connect
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
I'm setting up an instance with aws_kinesis_firehose as a source. I configures the tls settings with a valid certificate. When setting up the kinesis firehose to the HTTP endpoint, it fails to connect due to SSL communication error.
2024-02-05T12:57:33.928619Z DEBUG hyper::proto::h1::dispatch: error writing: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:33.928632Z DEBUG hyper::server::server::new_svc: connection error: error writing a body to connection: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:34.483435Z DEBUG hyper::proto::h1::conn: read eof
2024-02-05T12:57:34.483452Z DEBUG hyper::proto::h1::dispatch: error writing: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:34.483456Z DEBUG hyper::server::server::new_svc: connection error: error writing a body to connection: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:35.451551Z DEBUG sink{component_kind="sink" component_id=stdout component_type=console}: vector::utilization: utilization=0.004392145382787359
2024-02-05T12:57:35.686280Z DEBUG hyper::proto::h1::conn: read eof
2024-02-05T12:57:35.686296Z DEBUG hyper::proto::h1::dispatch: error writing: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:35.686300Z DEBUG hyper::server::server::new_svc: connection error: error writing a body to connection: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:36.093011Z DEBUG hyper::proto::h1::conn: read eof
I tried to use curl too, and got the following error:
curl https://****.****.com
curl: (35) LibreSSL/3.3.6: error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure
Configuration
sources:
logz:
type: aws_kinesis_firehose
address: 0.0.0.0:443
record_compression: gzip
store_access_key: true
access_keys:
- **************
decoding:
codec: json
tls:
crt_file: **************.crt
key_file: **************.key
key_pass: **************
enabled: true
verify_hostname: false
verify_certificate: true
transforms:
parse_logs:
type: remap
inputs:
- logz
source: |-
. = parse_json!(.message)
sinks:
stdout:
type: console
inputs:
- parse_logs
encoding:
codec: json
target: stdout
Version
vector 0.35.0 (x86_64-unknown-linux-gnu e57c0c0 2024-01-08 14:42:10.103908779)
Debug Output
2024-02-06T07:07:44.223897Z TRACE vector::topology::running: Spawning new sink. key=stdout
2024-02-06T07:07:44.223923Z INFO vector: Vector has started. debug="false" version="0.35.0" arch="x86_64" revision="e57c0c0 2024-01-08 14:42:10.103908779"
2024-02-06T07:07:44.223937Z INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2024-02-06T07:07:44.224020Z DEBUG transform{component_kind="transform" component_id=parse_logs component_type=remap}: vector::topology::builder: Synchronous transform starting.
2024-02-06T07:07:44.224034Z DEBUG source{component_kind="source" component_id=logz component_type=aws_kinesis_firehose}: vector::topology::builder: Source pump supervisor starting.
2024-02-06T07:07:44.224133Z DEBUG source{component_kind="source" component_id=logz component_type=aws_kinesis_firehose}: vector::topology::builder: Source pump starting.
2024-02-06T07:07:44.224153Z DEBUG source{component_kind="source" component_id=logz component_type=aws_kinesis_firehose}: vector::topology::builder: Source starting.
2024-02-06T07:07:44.224050Z DEBUG sink{component_kind="sink" component_id=stdout component_type=console}: vector::topology::builder: Sink starting.
2024-02-06T07:07:44.225246Z DEBUG sink{component_kind="sink" component_id=stdout component_type=console}: vector::utilization: utilization=0.03951496411321265
2024-02-06T07:07:44.225253Z TRACE vector: Beep.
2024-02-06T07:07:45.225356Z TRACE vector: Beep.
2024-02-06T07:07:46.225490Z TRACE vector: Beep.
2024-02-06T07:07:46.668932Z TRACE source{component_kind="source" component_id=logz component_type=aws_kinesis_firehose}: mio::poll: registering event source with poller: token=Token(139994061668736), interests=READABLE | WRITABLE
2024-02-06T07:07:46.669049Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-02-06T07:07:46.669291Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-02-06T07:07:46.731934Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-02-06T07:07:46.732001Z TRACE mio::poll: deregistering event source from poller
2024-02-06T07:07:46.732054Z TRACE hyper::proto::h1::conn: State::close_read()
2024-02-06T07:07:46.732058Z DEBUG hyper::proto::h1::conn: read eof
2024-02-06T07:07:46.732060Z TRACE hyper::proto::h1::conn: State::close_write()
2024-02-06T07:07:46.732067Z TRACE hyper::proto::h1::conn: State::close_read()
2024-02-06T07:07:46.732073Z TRACE hyper::proto::h1::conn: State::close_write()
2024-02-06T07:07:46.732077Z DEBUG hyper::proto::h1::dispatch: error writing: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-06T07:07:46.732089Z DEBUG hyper::server::server::new_svc: connection error: error writing a body to connection: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-06T07:07:47.225684Z TRACE vector: Beep.
2024-02-06T07:07:48.225832Z TRACE vector: Beep.
2024-02-06T07:07:49.225616Z TRACE vector: Beep.
2024-02-06T07:07:49.225633Z DEBUG sink{component_kind="sink" component_id=stdout component_type=console}: vector::utilization: utilization=0.00395152844882991
2024-02-06T07:07:50.225422Z TRACE vector: Beep.
2024-02-06T07:07:51.225687Z TRACE vector: Beep.
Example Data
No response
Additional Context
No response
References
No response
Hi @omers !
Thanks for filing this. Are you using a load balancer in-front of Vector? If so, could you provide more details about its configuration? I tried to reproduce this locally, but wasn't able to reproduce Vector not serving TLS.
Could you also try::
-
openssl s_client -connect ****.****.com
and provide the output -
curl -v https://****.****.com
and provide the output -
curl https://localhost
from the host running Vector and see if that works
Hi! Sorry for the late response. There is no LB between the client and the process.
The is the output of the openssl s_client:
0050C8D801000000:error:0A000410:SSL routines:ssl3_read_bytes:ssl/tls alert handshake failure:ssl/record/rec_layer_s3.c:861:SSL alert number 40
The curl output:
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, handshake failure (552):
* LibreSSL/3.3.6: error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure
* Closing connection
curl: (35) LibreSSL/3.3.6: error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure
The output from localhost:
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
Thanks @omers ! Those errors would may indicate that the certificate isn't valid for the hostname being used. Could you try curl --insecure
to see if curl
is able to make the request without requiring the certificate to be valid?
curl --insecure https://***-***.*****.com
curl: (35) LibreSSL/3.3.6: error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure
Hmm, that's odd. Is it serving TLS at all? Does curl http://...:443
work?
curl http://****.***-***.com:443
curl: (56) Recv failure: Connection reset by peer
Hmm. I'm not sure what's going on here then. Do you have a load balancer in the middle that could be misconfigured?
No load balancer, EC2 instance publicly exposed to 443 port.
Gotcha, I'm not sure then. I unfortunately haven't been able to reproduce this behavior locally; the aws_kinesis_firehose
source served TLS traffic fine :/