aws_kinesis_firehose source with tls enabled generate errors on connect

[omers]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I'm setting up an instance with aws_kinesis_firehose as a source. I configures the tls settings with a valid certificate. When setting up the kinesis firehose to the HTTP endpoint, it fails to connect due to SSL communication error.

2024-02-05T12:57:33.928619Z DEBUG hyper::proto::h1::dispatch: error writing: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:33.928632Z DEBUG hyper::server::server::new_svc: connection error: error writing a body to connection: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:34.483435Z DEBUG hyper::proto::h1::conn: read eof
2024-02-05T12:57:34.483452Z DEBUG hyper::proto::h1::dispatch: error writing: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:34.483456Z DEBUG hyper::server::server::new_svc: connection error: error writing a body to connection: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:35.451551Z DEBUG sink{component_kind="sink" component_id=stdout component_type=console}: vector::utilization: utilization=0.004392145382787359
2024-02-05T12:57:35.686280Z DEBUG hyper::proto::h1::conn: read eof
2024-02-05T12:57:35.686296Z DEBUG hyper::proto::h1::dispatch: error writing: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:35.686300Z DEBUG hyper::server::server::new_svc: connection error: error writing a body to connection: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-05T12:57:36.093011Z DEBUG hyper::proto::h1::conn: read eof

I tried to use curl too, and got the following error:

curl https://****.****.com
curl: (35) LibreSSL/3.3.6: error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure

Configuration

sources:
  logz:
    type: aws_kinesis_firehose
    address: 0.0.0.0:443
    record_compression: gzip
    store_access_key: true
    access_keys:
      - **************
    decoding:
      codec: json
    tls:
      crt_file: **************.crt
      key_file: **************.key
      key_pass: **************
      enabled: true
      verify_hostname: false
      verify_certificate: true

transforms:
  parse_logs:
    type: remap
    inputs:
      - logz
    source: |-
      . = parse_json!(.message)
sinks:
  stdout:
    type: console
    inputs:
      - parse_logs
    encoding:
      codec: json
    target: stdout

Version

vector 0.35.0 (x86_64-unknown-linux-gnu e57c0c0 2024-01-08 14:42:10.103908779)

Debug Output

2024-02-06T07:07:44.223897Z TRACE vector::topology::running: Spawning new sink. key=stdout
2024-02-06T07:07:44.223923Z  INFO vector: Vector has started. debug="false" version="0.35.0" arch="x86_64" revision="e57c0c0 2024-01-08 14:42:10.103908779"
2024-02-06T07:07:44.223937Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2024-02-06T07:07:44.224020Z DEBUG transform{component_kind="transform" component_id=parse_logs component_type=remap}: vector::topology::builder: Synchronous transform starting.
2024-02-06T07:07:44.224034Z DEBUG source{component_kind="source" component_id=logz component_type=aws_kinesis_firehose}: vector::topology::builder: Source pump supervisor starting.
2024-02-06T07:07:44.224133Z DEBUG source{component_kind="source" component_id=logz component_type=aws_kinesis_firehose}: vector::topology::builder: Source pump starting.
2024-02-06T07:07:44.224153Z DEBUG source{component_kind="source" component_id=logz component_type=aws_kinesis_firehose}: vector::topology::builder: Source starting.
2024-02-06T07:07:44.224050Z DEBUG sink{component_kind="sink" component_id=stdout component_type=console}: vector::topology::builder: Sink starting.
2024-02-06T07:07:44.225246Z DEBUG sink{component_kind="sink" component_id=stdout component_type=console}: vector::utilization: utilization=0.03951496411321265
2024-02-06T07:07:44.225253Z TRACE vector: Beep.
2024-02-06T07:07:45.225356Z TRACE vector: Beep.
2024-02-06T07:07:46.225490Z TRACE vector: Beep.
2024-02-06T07:07:46.668932Z TRACE source{component_kind="source" component_id=logz component_type=aws_kinesis_firehose}: mio::poll: registering event source with poller: token=Token(139994061668736), interests=READABLE | WRITABLE
2024-02-06T07:07:46.669049Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-02-06T07:07:46.669291Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-02-06T07:07:46.731934Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-02-06T07:07:46.732001Z TRACE mio::poll: deregistering event source from poller
2024-02-06T07:07:46.732054Z TRACE hyper::proto::h1::conn: State::close_read()
2024-02-06T07:07:46.732058Z DEBUG hyper::proto::h1::conn: read eof
2024-02-06T07:07:46.732060Z TRACE hyper::proto::h1::conn: State::close_write()
2024-02-06T07:07:46.732067Z TRACE hyper::proto::h1::conn: State::close_read()
2024-02-06T07:07:46.732073Z TRACE hyper::proto::h1::conn: State::close_write()
2024-02-06T07:07:46.732077Z DEBUG hyper::proto::h1::dispatch: error writing: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-06T07:07:46.732089Z DEBUG hyper::server::server::new_svc: connection error: error writing a body to connection: TLS handshake failed: error:0A0000C7:SSL routines:(unknown function):peer did not return a certificate:ssl/statem/statem_srvr.c:3726:
2024-02-06T07:07:47.225684Z TRACE vector: Beep.
2024-02-06T07:07:48.225832Z TRACE vector: Beep.
2024-02-06T07:07:49.225616Z TRACE vector: Beep.
2024-02-06T07:07:49.225633Z DEBUG sink{component_kind="sink" component_id=stdout component_type=console}: vector::utilization: utilization=0.00395152844882991
2024-02-06T07:07:50.225422Z TRACE vector: Beep.
2024-02-06T07:07:51.225687Z TRACE vector: Beep.

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hi @omers !

Thanks for filing this. Are you using a load balancer in-front of Vector? If so, could you provide more details about its configuration? I tried to reproduce this locally, but wasn't able to reproduce Vector not serving TLS.

Could you also try::

• openssl s_client -connect ****.****.com and provide the output
• curl -v https://****.****.com and provide the output
• curl https://localhost from the host running Vector and see if that works

[omers]

Hi! Sorry for the late response. There is no LB between the client and the process.

The is the output of the openssl s_client:

0050C8D801000000:error:0A000410:SSL routines:ssl3_read_bytes:ssl/tls alert handshake failure:ssl/record/rec_layer_s3.c:861:SSL alert number 40

The curl output:

* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, handshake failure (552):
* LibreSSL/3.3.6: error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure
* Closing connection
curl: (35) LibreSSL/3.3.6: error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure

The output from localhost:

curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

[jszwedko]

Thanks @omers ! Those errors would may indicate that the certificate isn't valid for the hostname being used. Could you try curl --insecure to see if curl is able to make the request without requiring the certificate to be valid?

[omers]

curl --insecure https://***-***.*****.com
curl: (35) LibreSSL/3.3.6: error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure

[jszwedko]

Hmm, that's odd. Is it serving TLS at all? Does curl http://...:443 work?

[omers]

curl http://****.***-***.com:443
curl: (56) Recv failure: Connection reset by peer

[jszwedko]

Hmm. I'm not sure what's going on here then. Do you have a load balancer in the middle that could be misconfigured?

[omers]

No load balancer, EC2 instance publicly exposed to 443 port.

[jszwedko]

Gotcha, I'm not sure then. I unfortunately haven't been able to reproduce this behavior locally; the aws_kinesis_firehose source served TLS traffic fine :/