relay
relay copied to clipboard
Relay becomes healthy with broken config and drops envelopes
Followup from #2428.
Relay with misconfiguration can become healthy and accept envelopes which will then be dropped due to misconfiguration.
E.g. a broken Kafka config:
kafka_config:
- { name: bootstrap.servers", value: "127.0.0.1:9092" }
Some misconfigurations can only be caught when actually processing an envelope. So we need to figure out a proper failure mode, possible solutions:
- Process dummy envelopes (ala smoke tests) during container startup
- Become unhealthy if an issue is detected
- Kill Relay
- ...
We also need to figure out if we can even properly catch and recognize these issues, e.g. can we differentiate between a broken Kafka configuration and just a Kafka node which is temporarily unavailable.