relay icon indicating copy to clipboard operation
relay copied to clipboard

Relay becomes healthy with broken config and drops envelopes

Open Dav1dde opened this issue 2 years ago • 0 comments

Followup from #2428.

Relay with misconfiguration can become healthy and accept envelopes which will then be dropped due to misconfiguration.

E.g. a broken Kafka config:

  kafka_config:
    - { name: bootstrap.servers", value: "127.0.0.1:9092" }

Some misconfigurations can only be caught when actually processing an envelope. So we need to figure out a proper failure mode, possible solutions:

  • Process dummy envelopes (ala smoke tests) during container startup
  • Become unhealthy if an issue is detected
  • Kill Relay
  • ...

We also need to figure out if we can even properly catch and recognize these issues, e.g. can we differentiate between a broken Kafka configuration and just a Kafka node which is temporarily unavailable.

Dav1dde avatar Nov 27 '23 09:11 Dav1dde