Cosmos DB trigger is not connecting to CosmosDB Emulator

[pejaab]

Hi Azure Functions Team, I have an Azure Functions App consisting of HttpTrigger Functions and CosmosDbTrigger Functions. Both kind of functions work fine and can connect against an actual CosmosDB instance. Unfortunately, when running against the CosmosDB emulator the CosmosDBTrigger Functions fail to connect against the CosmosDB emulator. The errors that I receive are of the following form:

[2022-01-15T18:17:06.601Z] Error indexing method 'Functions.feed'
[2022-01-15T18:17:06.601Z] Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.feed'. Microsoft.Azure.WebJobs.Extensions.CosmosDB: Cannot create Collection Information for Article in database ##-db with lease leases in database ##-db : The SSL connection could not be established, see inner exception. System.Net.Http: The SSL connection could not be established, see inner exception. System.Private.CoreLib: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch.

The settings of the app look like below:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=###;AccountKey=###;EndpointSuffix=core.windows.net",
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "DATABASE_CONNECTION_STRING": "AccountEndpoint=https://##.###.###.###:8081/;AccountKey=<DEFAULT_PRIMARY_KEY>"
   }
}

Though, the remaining HttpTrigger functions can communicate with the CosmosDB emulator via the SDK's CosmosClient. For this to work NODE_TLS_REJECT_UNAUTHORIZED=0 is set. But as the error looks like coming form the .NET functions tool itself not from any of our node application, it looks like this setting is not helping.

The CosmosDB emulator is setup as lined out in an answer to #1232 with the deviation that the local development is done on OSX, which is why the certificate import is different. And it looks like the imported certificate is ignored by .NET.

I believe a similar, if not same question was raised in #2024 was closed, but not solved. At least nothing I can find there helps me in resolving the described issues.

I'm not sure what else to try and would really appreciate any help at all :)

[michaelpeng36]

@ealsur Could you help out with this?

[ealsur]

You need to make sure the local environment has the required certificates from the Emulator, otherwise SSL validation will fail. There is no Trigger or SDK related issue here, it's just how SSL works, you need to know the certificate of the server in order for a client to connect. https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator?tabs=ssl-netstd21#macos-environment

[pejaab]

Thanks for pointing this out, but I took care of this already. After generating a certificate including the IP address via /GenCert. I added it to the local Keychain Access. And also set the trust settings according to your pointed out documentation. It's also pointed out in the above mentioned answer, that the certificate needs to be added. Unfortunately, this didn't change anything for the .NET functions. To me, it looks like it doesn't make a difference whether the certificate is installed or not. Just to be sure, if I understand (this documentation for .NET Core)[https://docs.microsoft.com/en-us/dotnet/standard/security/cross-platform-cryptography] correctly, then .NET is looking for certificates in the login.keychain on OSX. This is where I added and trusted the certificate. Is there anything further?

[ealsur]

Honestly not that I am aware, I am not an expert on the Emulator to know if there are any corner cases when using OSX + Windows image for the Emulator or if that is the right place for the certificates.

You are starting the Emulator allowing network access with the custom IP you are mentioning you are using on the certificate? The doc seems to mention localhost and not a custom IP though.

[pejaab]

The setup is more intricate, I guess. I'm starting the emulator on an external VM. Due to the new Apple Silicon, I'm not aware of a possibility of going down the route of local virtualization for now. Or do I miss something? CosmosDB is started with options /AllowNetworkAccess and /NoFirewall. The common name of the certificate is the mentioned localhost. If I generate a certificate with /GenCert, the certificate differs from the one without doing a /GenCert <IP_ADDRESS> by it including the given <IP_ADDRESS> as an additional DNS name. The Common Name of both certificates do not differ though. Writing this though I realize, that having a CN of localhost, will mean that the SSL validation cannot work. Is this correct? Is there any way around this? When SSH tunneling to the VM, this results in Request timed out. Is there any way to get this scenario to work?

[ealsur]

I checked the docs and there is no mention of /GenCert.

From what I see you need to start the emulator in the VM with the flags you have /AllowNetworkAccess and /NoFirewall, and then go to the Certificates in the VM and export the PFX and use that in the MacOS machine and connect to the public IP of the VM.

If not, you can reach to the Emulator team through this process: https://docs.microsoft.com/en-us/azure/cosmos-db/linux-emulator?tabs=ssl-netstd21#report-an-emulator-issue

[pejaab]

Thanks, the /GenCert command is documented here: https://docs.microsoft.com/en-us/azure/cosmos-db/emulator-command-line-parameters. I do exactly what you describe, but it doesn't do the trick, unfortunately.

Thanks for pointing out how to reach the Emulator team. I'll try my luck through the process you mentioned.

[ealsur]

@alrod This is not a Track2 Issue, this is related to the old Trigger leveraging V2 SDK and its purely a connectivity problem with the Cosmos DB emulator, not related to the trigger itself.

[macel94]

Hi @ealsur @michaelpeng36

i'm developing a simple but complex enough example to really try to understand how developing in containers would really be for me and my team in next projects. I'm really finding it hard to solve this simple thing of triggering my azfunction

i'm using everything with the latest packages and everything is public beside the secret i was required to create(not my choice) to be able to use azure service bus.

if you want to repro this bug i believe i can help, if you want to take a look here: https://github.com/MACEL94/advanced-azure-microservices-with-dotnet-for-developers/tree/9-manage-to-make-everything-work-locally-with-just-a-docker-compose-up

everything is on linux as i'm trying to develop on codespaces

thank you anyway for your help

[ealsur]

@MACEL94 There are no changes we can make on the Extension that should solve Emulator connection problems. That is a purely client + Emulator connectivity problem. The Extension or the SDK which is consumed underneath has no code referencing the Emulator at all and it should never have. Any connectivity problem between a client (SDK, Extensions, etc) and the Emulator needs to be investigated with the Emulator support: https://docs.microsoft.com/en-us/azure/cosmos-db/troubleshoot-local-emulator

[macel94]

Hi @ealsur I understand what you mean, just a question: do you think I have a way to override the default CosmosClient used by your implementation of "CosmosDBTrigger" through dependency injection?

Because if i could, i would be able to use the HttpFactoryOptions like i did here to ignore the certificate: https://github.com/MACEL94/advanced-azure-microservices-with-dotnet-for-developers/blob/9-manage-to-make-everything-work-locally-with-just-a-docker-compose-up/src/WisdomPetMedicine.Hospital.Infrastructure/PatientAggregateStore.cs#L27

thank you again for your time

[ealsur]

@MACEL94 We could expose an interface like we do for custom serialization: https://github.com/Azure/azure-webjobs-sdk-extensions/blob/dev/src/WebJobs.Extensions.CosmosDB/Config/ICosmosDBSerializerFactory.cs

Which in that case could be injected to customize the CosmosClientBuilder (just an idea). But really the point is that disabling SSL is just a shortcut to avoid dealing with installing the certificate and have it working as it should for local testing.

Another alternative would be to include a boolean in https://github.com/Azure/azure-webjobs-sdk-extensions/blob/dev/src/WebJobs.Extensions.CosmosDB/Config/CosmosDBOptions.cs that can be mapped to the JSON configuration (https://docs.microsoft.com/en-us/azure/cosmos-db/sql/how-to-configure-cosmos-db-trigger#configuring-the-connection-policy), like a disableSSL property. This would work across languages because the JSON config works on Functions on Java/Python/etc too.

[macel94]

Hi @ealsur "But really the point is that disabling SSL is just a shortcut to avoid dealing with installing the certificate" Yes, indeed. Because when I move in production, i don't expect to have the need to install a certificate while pointing to an actual instance of cosmosdb, so why should I do it in my local env?

Also I understand that I could probably get everything to work spinning up the emu container or the actual program via a script if I was on my windows machine, but the point is that I'm using codespaces so I would have to do something like: Spin up cosmos container

Either rebuild and download/trust the generated cert or build an image that starts with a bash script that only spins up my function after downloading the emulator cert and trusting it.

This seems overly complicated to me to do something that I'm not expected to be doing in an actual environment, but I'm open to suggestions.

I understand it's more a problem with the emulator forcing me to use the Https endpoint and not this trigger fault but I find strange that something this simple is becoming so hard ahah

Thank you anyway for your help!

[ealsur]

The big issue with adding this configuration is that if you accidentally push such a configuration to production, then you are now doing insecure requests.

[macel94]

The big issue with adding this configuration is that if you accidentally push such a configuration to production, then you are now doing insecure requests.

@ealsur yes that is absolutely true, but how does this differ or add value when I can bypass SSL when using the same cosmosdb in the same solution with my web APIs(because the CosmosClient here exposes this possibility to us Devs)?

Meanwhile in the same solution I can spawn a SQL server container in a similar way and connect to it without being forced to use SSL(yes, different persistence tech but it's still people from Microsoft following the same security guidelines while producing it, right?).

It seems to me that forcing SSL in the emulator, only in the cosmosdb + az function combo, causes more harm than anything else...

I can agree that it's a good thing to be secure by default, but I think we should be able to disable SSL here as we can do in other cases(API-cosmosdb connection).

I'm trying my best to demonstrate my team and bosses that "developing in Linux containers is the best thing ever, as well as the future" but if I can't get our most used services to work together I doubt I will be able to convince them :)

Thank you again for your precious time, if you read me

If you decide not to modify the trigger, I'd really like some guidance on how to get this part to work together with an actual example, and I'd be happy to help contributing to the docs if you need any help!

[ealsur]

Do you build the Functions mostly on C#? I think that a solution like the second one (add a config as a boolean value) could be the one that tackles this problem across languages.

[macel94]

Do you build the Functions mostly on C#? I think that a solution like the second one (add a config as a boolean value) could be the one that tackles this problem across languages.

Yes, I'm almost a C# only Dev as with blazor i can also do FE :)

Let me know if I can help in any way

Thanks again

[macel94]

@ealsur - @Ved2806 i understand this is not high priority, but are you willing to accept PRs? can i help?

[ealsur]

@MACEL94 I would avoid adding anything Emulator specific, because the Emulator is not meant to be used on production and it's just for local testing. And if this scenario is resolved in the future by a new Emulator version, then we'd have a flag that makes no sense.

Another potential idea could be to have an extension hook, like the ICosmosDBSerializerFactory:

https://gist.github.com/ealsur/d42abe66f04c092285452e7b4d5c5145

We could expose the HttpClientFactory and let the user configure the HTTP client, which includes customizing the SSL validation: https://learn.microsoft.com/en-us/azure/cosmos-db/local-emulator?tabs=ssl-netstd21#disable-ssl-validation

Still this is a potential danger if this accidentally ships to production.

[samuel-kogler-AP]

Any updates on this? Gateway mode works, but there seems to be no way to disable SSL verification in Direct mode (since it does not use the HttpClient).

And installing a root certificate is not an option, since it requires admin privileges on Windows (which I do not have).

[ealsur]

@samuel-kogler-AP there is nothing that can be done on Functions for this. This needs to be tackled on the Cosmos DB SDK, see: https://github.com/Azure/azure-cosmos-dotnet-v3/issues/4222

[ealsur]

With Extension 4.5.0 now you can use the ConnectionString to the emulator with the added flag to disable SSL.

Example:

AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==;IgnoreEndpointCertificate=true;