Kafka trigger in function app stops receiving -processing- messages randomly Exception detected on SDK Assembly

[nachoge]

Hello,

We have a set of funtions apps with kafka triggers despite they work perfectly most of the time we are experiencing isses regularly.

We have messages -we know they exist- received at the kafka server -a standalone kafka server- but functions do not process them -seems the client is not connected. The Azure monitors says the function aapp is running, so the only way to wake-up is by rebooting the function app.

Is there a know issue on this, or maybe a miss-configuration?

The platform: azure linux docker in premium plan with java sdk kafka extension

If you need the config used on function app side let me know

Thanks in advance

[nachoge]

Hi,

Continuing with this issue here are the details reported by insights.

Exception Properties

Message Broker: Coordinator load in progress Exception type: Confluent.Kafka.ConsumeException Exception type: Confluent.Kafka.ConsumeException Problem Id: Confluent.Kafka.ConsumeException at Confluent.Kafka.Consumer`2.Consume Assembly: Confluent.Kafka, Version=1.5.2.0, Culture=neutral, PublicKeyToken=12c514ca49093d1e Severity level: Error Telemetry type: exception Device type: PC SDK version azurefunctions: 3.2.0.0

Custom Properties

FormattedMessage: Consume error prop__{OriginalFormat}: Consume error LogLevel: Error Category: Host.Triggers.Kafka

Call Stack

Confluent.Kafka.ConsumeException: at Confluent.Kafka.Consumer2.Consume (Confluent.Kafka, Version=1.5.2.0, Culture=neutral, PublicKeyToken=12c514ca49093d1e) at Confluent.Kafka.Consumer2.Consume (Confluent.Kafka, Version=1.5.2.0, Culture=neutral, PublicKeyToken=12c514ca49093d1e) at Microsoft.Azure.WebJobs.Extensions.Kafka.KafkaListener`2.ProcessSubscription (Microsoft.Azure.WebJobs.Extensions.Kafka, Version=3.2.1.0, Culture=neutral, PublicKeyToken=null: /home/vsts/work/1/s/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Listeners/KafkaListener.cs:277)

Thanks

[TsuyoshiUshio]

I'm not sure why it stop processing, however, one thing you can do is enable debugging. https://tsuyoshiushio.medium.com/azure-functions-kafka-trigger-performance-tips-concept-and-architecture-ec94a31d8b93 Coud you see the debug section? If you can't find anything for the details logs, and broker side . This looks similar one. https://github.com/confluentinc/confluent-kafka-dotnet/issues/1090

[nachoge]

Hi TsuyoshiUshio,

I've setted up the debug. Right now with "broker,consumer" I did not set up fetch because the huge amount of traces. It's a matter of time to get info I will analize it -once the problem happens again- and I will give it to you

[shrohilla]

@nachoge are you still facing the same issue ??

[nachoge]

Hi @shrohilla,

Yes it still happends. It seems the trigger somehow goes zombie and stopps precessing messages. The internal azure monitoring does not detetcs any issue. To minimize this situation we have a monitoring mechanism which detecs the state and reboots the function.

Is there new version we could use? the installing process of the extension seems to be automatic so right now I don't know which versions we are using. By the way, how can I check the current triggret version?

Thanks

[shrohilla]

@nachoge if your app is in dotnet can you try with 3.6.0 released version of Kafka extension ??

[nachoge]

@shrohilla is not dotnet is Java. We are using Java SDK Kafka trigger extension

[shrohilla]

@nachoge not sure about your setup please share the sample code along annotation attributes, hosts.json for more details

[nachoge]

`@FunctionName("drop-functionapp-"+BuildConstants.BUILD_ENV+"-CF-OUT-ResolveOT") public void runOne( @KafkaTrigger( name = "kafkatriggerResolveOT", topic = "drop-"+BuildConstants.BUILD_ENV+"-CF-OUT-resolveOT", brokerList=BuildConstants.BUILD_KAFKA_BROKERS, consumerGroup="drop-"+BuildConstants.BUILD_ENV+"-CF-OUT-resolveOT", username=BuildConstants.BUILD_KAFKA_USERNAME, password=BuildConstants.BUILD_KAFKA_PASSWD, protocol=BrokerProtocol.SASLSSL, dataType="string", authenticationMode=BrokerAuthenticationMode.SCRAMSHA512, cardinality=Cardinality.ONE, sslCertificateLocation="/home/site/wwwroot/certificate/kafka.pem", sslCaLocation="/home/site/wwwroot/certificate/kafka.pem"

        ) String kafkaEventData,
        final ExecutionContext context) {

	// logic

    }`

host.json { "version":"2.0" }

is this enough? thanks

[shrohilla]

@nachoge are you running your function app using extensions.csproj file ??

[nachoge]

Hi, don't use it despite I have one outside the src

[shrohilla]

@nachoge as per my understanding of your current setup, please update the extensions.csproj file in the Package reference section <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Kafka" Version="3.6.0" />

run below command -- re-deploy with the latest kafka extension and would request you to share your feedback if this issue is appearing again.

[shrohilla]

No response from @nachoge seems this issue is resolved