Response status code does not indicate success: 409 (Conflict).

[paulo3339]

I have been using the Azure Lets Encrypt extension for several months to manage SSL certificates for several websites, without any issues. All of a sudden I'm unable to request any certificates on any of the app services and I get the following error. I'm running the latest version 0.9.6

Stack Trace:

[HttpRequestException: Response status code does not indicate success: 409 (Conflict).] System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() +224 LetsEncrypt.Azure.Core.Services.<Install>d__3.MoveNext() in D:\a\1\s\LetsEncrypt.SiteExtension.Core\Services\WebAppCertificateService.cs:55 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +99 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +58 LetsEncrypt.Azure.Core.<RequestAndInstallInternalAsync>d__17.MoveNext() in D:\a\1\s\LetsEncrypt.SiteExtension.Core\CertificateManager.cs:245 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +99 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +58 LetsEncrypt.SiteExtension.Controllers.<Install>d__7.MoveNext() in D:\a\1\s\LetsEncrypt-SiteExtension\Controllers\HomeController.cs:250 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +99 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +58 System.Web.Mvc.Async.TaskAsyncActionDescriptor.EndExecute(IAsyncResult asyncResult) +97 System.Web.Mvc.Async.<>c__DisplayClass8_0.<BeginInvokeAsynchronousActionMethod>b__1(IAsyncResult asyncResult) +17 System.Web.Mvc.Async.WrappedAsyncResult1.CallEndDelegate(IAsyncResult asyncResult) +10 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49 System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult) +32 System.Web.Mvc.Async.<>c__DisplayClass11_0.<InvokeActionMethodFilterAsynchronouslyRecursive>b__0() +58 System.Web.Mvc.Async.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2() +228 System.Web.Mvc.Async.<>c__DisplayClass7_0.<BeginInvokeActionMethodWithFilters>b__1(IAsyncResult asyncResult) +10 System.Web.Mvc.Async.WrappedAsyncResult1.CallEndDelegate(IAsyncResult asyncResult) +10 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49 System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult) +34 System.Web.Mvc.Async.<>c__DisplayClass3_6.<BeginInvokeAction>b__4() +35 System.Web.Mvc.Async.<>c__DisplayClass3_1.<BeginInvokeAction>b__1(IAsyncResult asyncResult) +100 System.Web.Mvc.Async.WrappedAsyncResult1.CallEndDelegate(IAsyncResult asyncResult) +10 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49 System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult) +27 System.Web.Mvc.<>c.<BeginExecuteCore>b__152_1(IAsyncResult asyncResult, ExecuteCoreState innerState) +11 System.Web.Mvc.Async.WrappedAsyncVoid1.CallEndDelegate(IAsyncResult asyncResult) +29 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49 System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult) +45 System.Web.Mvc.<>c.<BeginExecute>b__151_2(IAsyncResult asyncResult, Controller controller) +13 System.Web.Mvc.Async.WrappedAsyncVoid1.CallEndDelegate(IAsyncResult asyncResult) +22 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49 System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult) +26 System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.EndExecute(IAsyncResult asyncResult) +10 System.Web.Mvc.<>c.<BeginProcessRequest>b__20_1(IAsyncResult asyncResult, ProcessRequestState innerState) +28 System.Web.Mvc.Async.WrappedAsyncVoid1.CallEndDelegate(IAsyncResult asyncResult) +29 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49 System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult) +28 System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result) +9 System.Web.CallHandlerExecutionStep.InvokeEndHandler(IAsyncResult ar) +152 System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +126

[paulo3339]

To add to this, I have tried creating a completely new service principal with a new client secret, but I still get the same error. It might help if I knew what the error meant, but I can't seem to find any info about this error online that's related to lets encrypt.

[paulo3339]

We have located the issue ourselves, so I will include the details here for anybody that might have the same problem.

Looking at the activity log in Azure for the corresponding resource we found this error:

• Operation nameAdd or Update Certificate
• Time stampSun Jun 02 2019 20:40:26 GMT+0100 (BST)
• Event initiated byLetsEncrypt2
• Error codeResourceQuotaExceeded
• MessageCreating the resource of type 'Microsoft.Web/certificates' would exceed the quota of '800' resources of type 'Microsoft.Web/certificates' per resource group. The current resource count is '800', please delete some resources of this type before creating a new one.

So we've hit a limit on Azure. It seems to have been caused by the cleanup job failing and therefore the old certificates are not being removed.

[jbarranis]

@porky9 I am running into this issue now, so thanks for your post!

How did you clear out the old certificates to resolve this?

[paulo3339]

@jbarranis - We ran the cleanup job manually to identify which hostname was causing the issue. Then deleted the SSL binding for that hostname manually and re-ran the cleanup job to identify the next hostname. And repeated until they were all cleaned up.

But you can also view all expired certificates if you go to the app service and select 'TLS/SSL settings' and then select the 'Private Key Certificate' tab and select 'Expired' for status filter.

Then you can delete them manually, but I think it generally forces you to delete the binding first if the certificate is still bound to a hostname.

[jbarranis]

@porky9 I see. So do you have to delete the expired certificate? Or can that one just get renewed?

[paulo3339]

@jbarranis - I believe LetsEncrypt issues a new certificate each time and the cleanup job is supposed to delete the old ones. So yes, I would delete expired certificates. There shouldn't really be any if the cleanup job is working.

[paulo3339]

@jbarranis - I could be wrong about the way it's supposed to work though. That's what it seems to be doing with our install.

[jbarranis]

@porky9 - I gave that a try. I didn't see any 'Microsoft.Web/certificates' types in the activity log, so I don't know if I needed to do any manual cleanup. I went on ahead and removed the binding and deleted the expired certificate, then went through the steps of setting up a new one with the Azure Let's Encrypt extension. The last page gave me a 409 (Conflict) error. Have you run into this before?

[paulo3339]

@jbarranis Yes, that's the error we were getting. It turned out to be because we had hit the limit on Azure for the number of certificates you can have per resource group, which is 800. Have a look in the activity log in Azure and you will probably see a corresponding error.

[jbarranis]

@porky9 Hmm i'm not seeing it. I went to App Service > Activity Log and looked at the Resource Type column. Nothing is showing up in the past month, but there should be one from a few minutes ago. Am I looking in the right place? I don't know a whole lot about Devops stuff...

[paulo3339]

@jbarranis I generally click on the bell icon on the top menu bar and then click the link that says 'More events in the activity log'. That should show the activity log for the whole plan.

[jbarranis]

@porky9 It's not there either. Maybe I'm having a different issue. Also, I noticed after I deleted the expired cert, the web job is still running and now it's showing that the renewal is successful. I'm not even sure what it's renewing since that was the only SSL Cert I had...

[paulo3339]

@jbarranis - yep it could be a different issue. It's a pretty generic error.

It's probably just issued a new certificate for you.

You can check the healthy certificates in TLS/SSL settings. Then note down the thumbprint and make sure it matches the thumbprint of the certificate that's bound to the hostname. If it does then the certificate is healthy and it should be fine.

[jbarranis]

@porky9 That page is empty since I deleted that expired certificate. Is there a way to get it back, or do I need to resolve this 409 issue and create a new one?

[paulo3339]

@jbarranis you will need to resolve the 409 issue and create a new one

[jbarranis]

Hey @porky9 - This might be a little off topic now. But when I was trouble-shooting this, I decided to start the set up over from scratch. In App Settings > Configuration > Application Settings heading, I had a bunch of things like:

letsencrypt:ClientId letsencrypt:ClientSecret ...

When I started over, those got cleared out. Then I went to Extensions and created a whole new SSL cert. I got an error:

System.Exception: The Lets Encrypt ACME server was probably unable to reach http://domain.com/.well-known/acme-challenge/token

More googling, someone suggested I needed letsencrypt:WebRootPath (value D:\home\site\wwwroot\). That still gave me that error above. Then I changed the value to D:\site\wwwroot\, and that gave me the 409 error.

Could this be something incorrect in my settings? It was working the first time I set it up, so I'm not sure why it would break now.

[jbarranis]

Okay, I finally figured it out. The errors from letsencrypt extension were not helpful.

[sjkp]

@jbarranis feel free to share what the solution to your problem was, and if you have any suggestions to how the errors from the extension could have helped you troubleshoot the issue faster.

[jbarranis]

@sjkp sure.

I narrowed down the 409 error being due to an incorrect configuration in the Application Settings. The letsencrypt extension doesn't add the letsencrypt:WebRootPath to the settings when you go through and create the SSL cert. When I put in the wrong path, it gave me a 409 error.

Browsing through the Kudu files, I'm seeing that it should be D:\home\site\wwwroot\, but as mentioned above, it led to another error. Two things finally worked:

1) Make sure the `HTTPS Only` is turned OFF while recreating the SSL cert. I got a 400 error that was unclear when I had this swithed on.

2) Add the correct path and to ensure the letsencrypt:WebRootPath points to the right directory,

Note: I found this old issue https://github.com/sjkp/letsencrypt-siteextension/issues/239 and followed the solution by xt0rted and completely uninstalled and reinstalled the extension at one point, so I'm not 100% sure if this helped fix the problem. I would check the above 2 things first before uninstalling/reinstalling.

[jbarranis]

@sjkp Unfortunately, I don't remember the exact error message, but it could be helpful to add a note about the HTTPS Only setting when creating the cert. Also, there are substatus error codes for IIS. Perhaps a 403.1 or something else might be more suitable.

[JimBobSquarePants]

I'm suddenly seeing this with 1.0.5. I haven't any issues previously.

Both settings mentioned above are configured

1. Make sure the HTTPS Only is turned OFF while recreating the SSL cert. I got a 400 error that >was unclear when I had this swithed on.

2. Add the correct path and to ensure the letsencrypt:WebRootPath points to the right directory.

I've also uninstalled and reinstalled the extension (though not changed any application settings)

[HttpRequestException: Response status code does not indicate success: 409 (Conflict).]
   System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() +211
   LetsEncrypt.Azure.Core.<PutFile>d__11.MoveNext() in D:\a\1\s\LetsEncrypt.SiteExtension.Core\KuduRestClient.cs:71
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   LetsEncrypt.Azure.Core.Services.<WriteFile>d__7.MoveNext() in D:\a\1\s\LetsEncrypt.SiteExtension.Core\Services\KuduFileSystemAuthorizationChallengeProvider.cs:53
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   LetsEncrypt.Azure.Core.Services.<EnsureWebConfig>d__5.MoveNext() in D:\a\1\s\LetsEncrypt.SiteExtension.Core\Services\KuduFileSystemAuthorizationChallengeProvider.cs:37
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   LetsEncrypt.Azure.Core.Services.<Authorize>d__5.MoveNext() in D:\a\1\s\LetsEncrypt.SiteExtension.Core\Services\BaseHttpAuthorizationChallengeProvider.cs:55
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   LetsEncrypt.Azure.Core.Services.<RequestCertificate>d__5.MoveNext() in D:\a\1\s\LetsEncrypt.SiteExtension.Core\Services\AcmeService.cs:55
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   LetsEncrypt.Azure.Core.<RequestInternalAsync>d__14.MoveNext() in D:\a\1\s\LetsEncrypt.SiteExtension.Core\CertificateManager.cs:206
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   LetsEncrypt.Azure.Core.<RequestAndInstallInternalAsync>d__15.MoveNext() in D:\a\1\s\LetsEncrypt.SiteExtension.Core\CertificateManager.cs:230
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   LetsEncrypt.SiteExtension.Controllers.<Install>d__7.MoveNext() in D:\a\1\s\LetsEncrypt-SiteExtension\Controllers\HomeController.cs:271
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   System.Web.Mvc.Async.TaskAsyncActionDescriptor.EndExecute(IAsyncResult asyncResult) +92
   System.Web.Mvc.Async.<>c__DisplayClass8_0.<BeginInvokeAsynchronousActionMethod>b__1(IAsyncResult asyncResult) +22
   System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult) +42
   System.Web.Mvc.Async.<>c__DisplayClass11_0.<InvokeActionMethodFilterAsynchronouslyRecursive>b__0() +80
   System.Web.Mvc.Async.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2() +387
   System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult) +42
   System.Web.Mvc.Async.<>c__DisplayClass3_6.<BeginInvokeAction>b__4() +42
   System.Web.Mvc.Async.<>c__DisplayClass3_1.<BeginInvokeAction>b__1(IAsyncResult asyncResult) +188
   System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult) +38
   System.Web.Mvc.<>c.<BeginExecuteCore>b__152_1(IAsyncResult asyncResult, ExecuteCoreState innerState) +26
   System.Web.Mvc.Async.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) +73
   System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult) +52
   System.Web.Mvc.Async.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) +39
   System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult) +38
   System.Web.Mvc.<>c.<BeginProcessRequest>b__20_1(IAsyncResult asyncResult, ProcessRequestState innerState) +40
   System.Web.Mvc.Async.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) +73
   System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult) +38
   System.Web.CallHandlerExecutionStep.InvokeEndHandler(IAsyncResult ar) +234
   System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +169

[JimBobSquarePants]

Ok... So I've figured out the cause in my case.

Several of the files in my site\wwwroot\.well-known\acme-challenge folder had somehow been marked readonly which was leading to the following error 409 conflict: could not write to local resource.

You can check and alter the permissions of a file using the attrib command through the Kudu console. Here's a fantastic blog post showing how to do that.

https://azureappservices.blogspot.com/2018/06/unable-to-editdelete-files-through-kudu.html

I ran the following attrib -r *.* to remove the readonly permissions from all the files and then deleted everything in the folder. Running the lets encrypt site extension again installed the certificate without issue.

[thinkbeforecoding]

I managed to fix it in my azure function by setting the app config WEBSITE_RUN_FROM_PACKAGE to 0

When it was 1, Kudu refused the editing.