Health Check / Readiness

[PSanetra]

I have not found any way to validate the health and readiness of the Kafka Listeners. Did I miss something?

Specifically we are planning to deploy the .NET application, which uses the WebJob SDK and this Kafka extension, as container in an AKS cluster. Therefore it would be nice to have some mechanism to check if the application is really ready to process messages (at least checking if it can connect to the Kafka broker at all.).

For example we got the following error logs due to some misconfiguration, but we really needed to check to logs to catch this issue.

[...]
info: Microsoft.Azure.WebJobs.Hosting.OptionsLoggingService[0]
      KafkaOptions
      {
        "SessionTimeoutMs": 30000,
        "MaxBatchSize": 64,
        "AutoCommitIntervalMs": 200,
        "AutoOffsetReset": 1,
        "MetadataMaxAgeMs": 180000,
        "SocketKeepaliveEnable": true,
        "SubscriberIntervalInSeconds": 1,
        "ExecutorChannelCapacity": 1,
        "ChannelFullRetryIntervalInMs": 50
      }
warn: Host.Triggers.Kafka[0]
      Libkafka: [thrd:app]: Broker name "SASL_SSL://**redacted**.westeurope.azure.confluent.cloud:9092" parse error: protocol "SASL_SSL" does not match security.protocol setting "plaintext"
info: Host.Triggers.Kafka[0]
      FunctionExecutor Loaded: MultipleItemFunctionExecutor
info: Host.Startup[0]
      Job host started
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
01/18/23 06:19:27.239 PM [1|43] [info] *** CallTarget_RewriterCallback() Finished: Confluent.Kafka.Consumer`2.Consume() [IsVoid=0, IsStatic=0, IntegrationType=Datadog.Trace.ClrProfiler.AutoInstrumentation.Kafka.KafkaConsumerConsumeIntegration, Arguments=1]
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app
fail: Host.Triggers.Kafka[0]
      No brokers configured

It would be better to mark the pod as not ready when the Kafka Trigger can not connect to the broker.