KafkaTrigger fails to load SslCertificateLocation and SslKeyLocation

[Irate-Walrus]

• On initialization of KafkaTrigger in Azure Premium Function, client public certificate and client secret will fail to load unless reference by absolute path within the trigger configuration.
• I can confirm that all files are present in the same directory as server_ca.crt. It appears that for some reason Libkafka is not given the full path to the client.crt or client.pem and hence fails to load them.
• The certificate and key successfully load when using the Kafka output binding.
• This issue occurs with release 3.3.2 available from https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.Kafka/

2022-04-19T03:09:32.209 [Debug] Librdkafka initialization: loading librdkafka from C:\home\site\wwwroot\bin\runtimes\win-x86\native\librdkafka.dll
2022-04-19T03:09:32.286 [Debug] Found SslCaLocation in C:\home\site\wwwroot\server_ca.crt
2022-04-19T03:09:32.306 [Debug] Libkafka: [thrd:app]: librdkafka built with OpenSSL version 0x1000211f
2022-04-19T03:09:32.306 [Debug] Libkafka: [thrd:app]: Loading CA certificate(s) from file C:\home\site\wwwroot\server_ca.crt
2022-04-19T03:09:32.306 [Debug] Libkafka: [thrd:app]: Loading public key from file client.crt
2022-04-19T03:09:32.307 [Error] Libkafka: [thrd:app]: .\crypto\bio\bss_file.c:406: error:02001002:system library:fopen:No such file or directory: fopen('client.crt','rb')
2022-04-19T03:09:32.342 [Error] Libkafka: [thrd:app]: .\crypto\bio\bss_file.c:408: error:20074002:BIO routines:FILE_CTRL:system lib
2022-04-19T03:09:32.385 [Error] A host error has occurred during startup operation '91393c74-7d66-4aef-ab74-ce5efab1c16f'.System.InvalidOperationException : ssl.certificate.location failed: .\ssl\ssl_rsa.c:701: error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system libat Confluent.Kafka.Impl.SafeKafkaHandle.Create(RdKafkaType type,IntPtr config,IClient owner)at Confluent.Kafka.Consumer`2..ctor(ConsumerBuilder`2 builder)at Confluent.Kafka.ConsumerBuilder`2.Build()at Microsoft.Azure.WebJobs.Extensions.Kafka.KafkaListener`2.CreateConsumer() at /home/vsts/work/1/s/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Listeners/KafkaListener.cs : 112at Microsoft.Azure.WebJobs.Extensions.Kafka.KafkaListener`2.<.ctor>b__23_0() at /home/vsts/work/1/s/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Listeners/KafkaListener.cs : 79at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication,Boolean useDefaultConstructor)at System.Lazy`1.CreateValue()at System.Lazy`1.get_Value()at Microsoft.Azure.WebJobs.Extensions.Kafka.KafkaListener`2.CreateTopicScaler() at /home/vsts/work/1/s/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Listeners/KafkaListener.cs : 117at Microsoft.Azure.WebJobs.Extensions.Kafka.KafkaListener`2.<.ctor>b__23_1() at /home/vsts/work/1/s/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Listeners/KafkaListener.cs : 80at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication,Boolean useDefaultConstructor)at System.Lazy`1.CreateValue()at System.Lazy`1.get_Value()at Microsoft.Azure.WebJobs.Extensions.Kafka.KafkaListener`2.GetMonitor() at /home/vsts/work/1/s/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Listeners/KafkaListener.cs : 403at Microsoft.Azure.WebJobs.Host.Listeners.HostListenerFactory.RegisterScaleMonitor(IListener listener,IScaleMonitorManager monitorManager) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\HostListenerFactory.cs : 113at async Microsoft.Azure.WebJobs.Host.Listeners.HostListenerFactory.CreateAsync(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\HostListenerFactory.cs : 73at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at async Microsoft.Azure.WebJobs.Host.Listeners.ListenerFactoryListener.StartAsyncCore(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\ListenerFactoryListener.cs : 45at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at async Microsoft.Azure.WebJobs.Host.Listeners.ShutdownListener.StartAsync(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\ShutdownListener.cs : 29at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at async Microsoft.Azure.WebJobs.JobHost.StartAsyncCore(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\JobHost.cs : 97at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at async Microsoft.Azure.WebJobs.Script.ScriptHost.StartAsyncCore(CancellationToken cancellationToken) at /_/src/WebJobs.Script/Host/ScriptHost.cs : 270at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at async Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at async Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(ScriptHostStartupOperation activeOperation,Int32 attemptCount,JobHostStartupMode startupMode) at /_/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs : 309

[shrohilla]

Please share your configuration with functions.json & other related details

[Irate-Walrus]

📝 I did have to change some values for confidentiality, but nothing meaningful.

function.json

{
"generatedBy": "Microsoft.NET.Sdk.Functions.Generator-4.1.0",
"configurationSource": "attributes",
"bindings": [
    {
    "type": "kafkaTrigger",
    "sslCaLocation": "server.crt",
    "sslCertificateLocation": "client.crt",
    "sslKeyLocation": "client.key",
    "protocol": "ssl",
    "consumerGroup": "consumerGroup",
    "topic": "%Topic%",
    "brokerList": "%BrokerList%",
    "authenticationMode": "notSet",
    "lagThreshold": 1000,
    "name": "events"
    }
],
"disabled": false,
"scriptFile": "../bin/KafkaConnector.dll",
"entryPoint": "KafkaSolution.KafkaController.Subscriber"
}

host.json

{
  "extensions": {
  },
  "logging": {
    "logLevel": {
      "default": "Warning",
      "Function": "Debug",
      "Host.Triggers.Kafka": "Warning"
    },
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  },
  "version": "2.0"
}

Working function.json

{
"generatedBy": "Microsoft.NET.Sdk.Functions.Generator-4.1.0",
"configurationSource": "attributes",
"bindings": [
    {
    "type": "kafkaTrigger",
    "sslCaLocation": "server.crt",
    "sslCertificateLocation": "%HOME%\\site\\wwwroot\\client.crt",
    "sslKeyLocation": "%HOME%\\site\\wwwroot\\client.key",
    "protocol": "ssl",
    "consumerGroup": "consumerGroup",
    "topic": "%Topic%",
    "brokerList": "%BrokerList%",
    "authenticationMode": "notSet",
    "lagThreshold": 1000,
    "name": "events"
    }
],
"disabled": false,
"scriptFile": "../bin/KafkaConnector.dll",
"entryPoint": "KafkaSolution.KafkaController.Subscriber"
}

[shrohilla]

Can you share the details of below two points:-

1. Can you add the complete relative path of the sslCertificateLocation & sslCaLocation and retry
2. Can you check "Copy to Output Directory" configuration value, I am wondering if certificate is copied in bin directory

[Irate-Walrus]

Project Configuration for CopyToOutputDirectory

<ItemGroup>
    <None Update="server_ca.crt">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="client.crt">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="client.key">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
</ItemGroup>

wwwroot

[Irate-Walrus]

Some paths I've tried:

Path	Azure	Local Debug (self-explanatory)
"%HOME%\\site\\wwwroot\\server.crt"	✔️	❌
"D:\\home\\site\\wwwroot\\server.crt"	✔️	❌
".\\server_ca.crt"	❌	✔️
"server_ca.crt"	❌	✔️

• I might add I am using both a Kafka Output Binding and a KafkaTrigger in two different function in the same premium function app.
• The Kafka Output Binding works with all paths including "server_ca.crt".

📝 I use server_ca.crt as an example here it will typically fail in order of file load:

1. server_ca.crt
2. client.crt
3. client.key

[MichalLechowski]

For WebJob (same nuget) I just use the same kinda approach, I have the certs in the root of my app and I use the same nuget version.

SslCaLocation = "rootcert.pem",
SslCertificateLocation = "clientcert.pem",
SslKeyLocation = "privatekey.pem",

This just works all right. What also works is when I set it up like this e.g.:

SslCaLocation = "pathInRootDir//rootcert.pem"

But it doesn't work with only single /

I use Docker to build and deploy to AKS on Linux nodes. But it also works locally since I don't see any certificates issues when running in Docker Desktop or just locally as ..NET 5 Console App (WebJob) and when I set a wrong file or a path, it throws an error. I have also had some kafka subs running as ACI in Azure, it works, too.

One difference I see is that I have AuthMode set to Plain but it shouldn't matter.

[Irate-Walrus]

Hi @MichalLechowski, Thank you for the information. I do believe the issue is platform dependent as the filename works correctly locally. The function is running .NET 6 on a Windows Premium Function. Deployment is done via a local build and upload as a package.

[MichalLechowski]

Okay.

Can you:

1. Share your function code, the method with Input Kafka trigger? You can skip the body, just the whole declaration with attributes and such. If you have some passwords there, obviously remove them.
2. Can you go to your Azure function then go to Advanced Tools then click Ok to open Kudu and then you have access to your function file system and check if actually the cert file is where you think it is? You need to open debug console in Kudu (CMD or PowerShell, doesn't matter) and there you have access to file system over simple GUI. Then you'll see your typical sites/wwwroot and such.

[Irate-Walrus]

@MichalLechowski ,

1. Share your function code, the method with Input Kafka trigger? This is the broken configuration of the trigger, just a reflection of the function.json in above comment

2. Can you go to your Azure function then go to Advanced Tools then click Ok to open Kudu etc. Can confirm they are in D:\home\site\wwwroot as previously shown in above comment

[MichalLechowski]

Okay, I've setup a basic azf (.net 6 isolated) with kafka trigger and deployed to Azure Function App running on Windows. It looks like the problem is that you think that execution context is the same directory as dlls of your built project (basically wwwroot in this case) but it's probably not and you always need to set it up with a proper path, that's why "%HOME%\site\wwwroot\server.crt" works just fine for you and that is why plain filename works when running locally since locally it looks for the file in bin/Release/ directory (for Release environment) where all your dlls and certificates are.

You could play with it and somehow dynamically build the path but I don't it's worth it. Just use environment variables in Function App and in local.settings.json and set proper paths for diffrent environments. When running locally it'll use local.settings.json and when running in Azure, it'll use the ones setup in Configuration part of Function App in Azure.

That is not a problem with the kafka extension, just how Functions work in Azure. Maybe it'd be different for Azure Function running on Linux, I don't know but you can try if you have time.

That is also why there is ExecutionContext class available you can add in your function method signature to get the directory during runtime but you cannot use it in an attribute value, obviously.

[Irate-Walrus]

Thanks for the sanity check, it's good to hear you've experienced the same issue.

You could play with it and somehow dynamically build the path but I don't it's worth it.

Just regarding this, I thought the package already did in the AzureFunctionsFileHelper. AzureFunctionsFileHelper is called in KafkaProducerFactory. I believe this is the reason why I don't have this issue when using the Kafka output binding.

// KafkaProducerFactory.cs
public ProducerConfig GetProducerConfig(KafkaProducerEntity entity)
{
    if (!AzureFunctionsFileHelper.TryGetValidFilePath(entity.Attribute.SslCertificateLocation, out var resolvedSslCertificationLocation))
    {
        resolvedSslCertificationLocation = entity.Attribute.SslCertificateLocation;
    }

However, looking into this further, the KafkaTriggerAttributeBindingProvider doesn't actually use the helper, and therefore breaks without the file path.

// KafkaTriggerAttributeBindingProvider.cs
// CreateConsumerConfiguration
consumerConfig.SaslPassword = this.config.ResolveSecureSetting(nameResolver, attribute.Password);
consumerConfig.SaslUsername = this.config.ResolveSecureSetting(nameResolver, attribute.Username);
consumerConfig.SslKeyLocation = this.config.ResolveSecureSetting(nameResolver, attribute.SslKeyLocation);
consumerConfig.SslKeyPassword = this.config.ResolveSecureSetting(nameResolver, attribute.SslKeyPassword);
consumerConfig.SslCertificateLocation = this.config.ResolveSecureSetting(nameResolver, attribute.SslCertificateLocation);
consumerConfig.SslCaLocation = this.config.ResolveSecureSetting(nameResolver, attribute.SslCaLocation);

What I do still find confusing is why the server_ca.crt is found as seen in my initial comment and why the file loading behavior is different between the Kafka Output and the Kafka Trigger:

2022-04-19T03:09:32.209 [Debug] Librdkafka initialization: loading librdkafka from C:\home\site\wwwroot\bin\runtimes\win-x86\native\librdkafka.dll
2022-04-19T03:09:32.286 [Debug] Found SslCaLocation in C:\home\site\wwwroot\server_ca.crt
2022-04-19T03:09:32.306 [Debug] Libkafka: [thrd:app]: librdkafka built with OpenSSL version 0x1000211f
2022-04-19T03:09:32.306 [Debug] Libkafka: [thrd:app]: Loading CA certificate(s) from file C:\home\site\wwwroot\server_ca.crt
2022-04-19T03:09:32.306 [Debug] Libkafka: [thrd:app]: Loading public key from file client.crt
2022-04-19T03:09:32.307 [Error] Libkafka: [thrd:app]: .\crypto\bio\bss_file.c:406: error:02001002:system library:fopen:No such file or directory: fopen('client.crt','rb')

[shrohilla]

This seems a bug, we will look into this

[MichalLechowski]

That's weird, indeed. One thing you can check is what happens (just as an experiment) when you set it up manually instead of using an attribute. Configured manually and triggered with TimeTrigger, for instance, based on e.g. this: https://github.com/Azure/azure-functions-kafka-extension/blob/dev/samples/dotnet/ConsoleConsumer/Program.cs

I have been wondering if that might have anything to do with permissions. Maybe the file exists but the app has no permissions to access it. Just a thought.

[shrohilla]

@Irate-Walrus Thanks alot for reporting this bug, we had fixed this from 3.6.0 release