Invalid Logstash pipeline configurations are not detected

[lpeter91]

Since automatic config reloading (config.reload.automatic) is implicitly enabled by the operator, invalid pipeline configurations don't cause the container to exit, so such errors are never detected (the pod is running and the logstash resource reports green health), leaving users with the false impression that everything is in order.

Quite simple to reproduce:

1. Spin up Logstash with any invalid configuration, for example:

cat <<'EOF' | kubectl apply -f -
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  count: 1
  version: 8.17.4
  pipelines:
    - pipeline.id: main
      config.string: |
        invalid
EOF

2. Check logs to verify that we have an issue: kubectl logs -f quickstart-ls-0
3. Verify that the error is not detected and "green" health is reported: kubectl describe logstashes.logstash.k8s.elastic.co quickstart

I guess the proper fix would be to implement health checks using the Logstash API. As a workaround one can explicitly disable config reloading, and annotate the pod with the hash of the pipeline config to recreate on changes.

[pebrc]

I am on the fence whether this should be triaged as bug or enhancement. I'm tending towards enhancement because the health indicator on the custom resources as has traditionally in ECK mapped to container health/readiness and not necessarily taken application health into account.

[lpeter91]

Well yeah, it's a bit blurry line. However I'm under the impression that the health of the elasticsearch resource (for example) also considers the actual cluster health. And for other componens a severely invalid config tends to crash the container, so from an end user perspective such errors do bubble up into the resource health.

But since there's a workaround for this problem I'm fine with calling it an enchanment.