sjkp
Functions.RenewCertificate always fails
I'm a new user of this extension and I have been able to install a new certificate for my site, however I've noticed in Kudu that Functions.RenewCertificate always fails with the following stack trace:
Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.RenewCertificate ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error). at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at LetsEncrypt.SiteExtension.Core.CertificateManager.<RenewCertificate>d__8.MoveNext() in J:\Projects\letsencrypt-siteextension\LetsEncrypt.SiteExtension.Core\CertificateManager.cs:line 82 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at LetsEncrypt.SiteExtension.Functions.<RenewCertificate>d__1.MoveNext() in J:\Projects\letsencrypt-siteextension\LetsEncrypt.SiteExtension.WebJob\Functions.cs:line 61 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker``1.<InvokeAsync>d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithWatchersAsync>d__31.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__2c.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__13.MoveNext() --- End of inner exception stack trace --- at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__13.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__1.MoveNext()
This appears to be the same point of failure encountered by dajbych in this issue.
I'm using v0.6.17 of the extension, the current latest version. I've narrowed the issue down to this Azure REST API call failing:
GET https://management.azure.com/subscriptions/<my subscription id>/providers/Microsoft.Web/certificates?api-version=2016-03-01
It fails with an HTTP 500 Internal Server Error and includes the message "Could not get certificate with thumbprint 3E2BF7F2031B96F38CE6C4D8A85D3E2D58476A0F". The thing is, I can't see that certificate anywhere in the Azure portal - this isn't the same thumbprint as the certificate that Let's Encrypt extension installed. This subscription only has a single resource group and, if I use the REST API enumerate the certificates in the resource group, I only see the certificate that Let's Encrypt extension installed.
I suspect that my problem is somehow related to my Azure subscription, but I'm not sure where to explore next. This may not be something you can help with, but if you can give me a pointer, I'd appreciate it!
@dariom thanks for trying the extension and reporting your findings.
I have an idea to what could be going on. Are you on a shared azure subscription, where you don't have access to everything? Because maybe it is permission issue with the https://management.azure.com/subscriptions/<my subscription id>/providers/Microsoft.Web/certificates?api-version=2016-03-01 endpoint that the service principal doesn't have access to list all certificates available in the subscription, and it then fails with a rather obscure message.
Ideally I would like not to use that endpoint and instead use (https://resources.azure.com/subscriptions/<my-sub>/resourceGroups/LetsEncrypt.Azure/providers/Microsoft.Web/certificates), but in the case where people move web apps between resource groups, I had no other way to find the certificates than to use the subscription scoped endpoint.
If you could also try to use the https://resources.azure.com and see if that also fails for you, you should be able to browse the same endpoints (as your self) as the site extension is trying.
@sjkp many thanks for your response.
I'm not on a shared Azure subscription - it's my personal one. My account is a subscription owner.
When I use the Let's Encrypt service principal, I can query https://resources.azure.com/subscriptions/<my-sub>/resourceGroups/<my-rg>/providers/Microsoft.Web/certificates and retrieve the correct certificate.
Following your advice I tried using https://resources.azure.com as myself and I'm able to replicate the same error described above: "Could not get certificate with thumbprint 3E2BF7F2031B96F38CE6C4D8A85D3E2D58476A0F".
I also tried temporarily granting subscription owner permissions to the Let's Encrypt service principal but unfortunately that didn't change anything.
I just googled the thumbprint, and it comes up quite a lot which is kinda strange unless it is an intermediate or root certificate. Do you have other certs installed, some which might have intermediate certificates that have not been installed, and thus the certificate chain cannot be verified. I'm not sure if this is the reason, but that is the only I can think of.
It would be interesting to open a support case with MS on it. It might be a bug in the ARM API, because in my opinion you should always be able to list your certificates using https://management.azure.com/subscriptions/<my subscription id>/providers/Microsoft.Web/certificates?api-version=2016-03-01
Another thing you could try is to use the ARMClient (https://github.com/projectkudu/ARMClient) and call the endpoint using different api-version to see if it makes any difference using an older version.
These are the supported versions
"apiVersions": [
"2016-03-01",
"2015-08-01-preview",
"2015-08-01",
"2015-07-01",
"2015-06-01",
"2015-05-01",
"2015-04-01",
"2015-02-01",
"2014-11-01",
"2014-06-01",
"2014-04-01-preview",
"2014-04-01"
]
FYI - I'm having the same problem. I upgraded from an older version to use the "cleanup" code that was added but now i get the same error. Service Principal was created based on the tutorial.
Seem's odd it would just break...and i'm not sure i fully understand the reasoning above. Is there something that need to happen with the service principal?
The old versions didn't use this API, which I why this wasn't a problem.
The reason why I changed to this API. Was that people who moved their web app to another resource group wouldn't be able to assign certificates anymore, as that would not work with the old API. But I guess I should do a fallback in case the new way of doing things doesn't work.
so i tried to create a new service principal to see if that would fix it but unfortunately not. I'm trying to roll back to an older version that i know was working but seem to be having a hell of a time finding any documentation on if that's even possible? Do you have any idea how to roll back to a previous site extension manually?
@sjkp thanks for the further feedback on this issue. It was a good idea to search for the thumbprint - I should have thought of that! :)
The thumbprint belongs to the StartCom Certification Authority certificate. It's not an intermediate certificate. I used StartCom before switching to Let's Encrypt (because of some security concerns). I previously had installed a StartCom issued, free SSL certificate and an intermediate certificate. I've since deleted the certificate but perhaps there is some sort of problem with the certificate chain. I think the best path forwards is opening a support case with Microsoft. I'll feedback my progress with that if you're interested.
Incidentally, I tried the ARMClient and tested all versions of the API which yielded the exact same result :(
I must thank you for your insights and assistance, you've been a great help!
@dariom We have engaged the product team. There seems to be an inconsistent certificate infos within our system leading to the issue. Will update.
@suwatch I've tried calling the REST API again (GET https://management.azure.com/subscriptions/<my subscription id>/providers/Microsoft.Web/certificates?api-version=2016-03-01) and it works! The API returns a HTTP 200 response code :)
Can you elaborate on the issue and what's changed?
More importantly, are the certificates returned expected (no more no less)?
It is an infrastructure issue where our records were inconsistent. We have hand fixed it to unblock you. We will address them overall in the next release (ETA 3-4 weeks).
Two certificates are returned. A Let’s Encrypted certificate and a StartCom intermediate certificate (thumbprint 3E2BF7F2031B96F38CE6C4D8A85D3E2D58476A0F).
The latter certificate is somewhat of a surprise. The id attribute for the certificate includes the name of a resource group that I’ve deleted previously (“Default-Web-EastAsia”). I would have expected the intermediate certificate to have been deleted along with the resource group.
@suwatch @dariom nice to see MS helping out
@ai-fwd In theory you can just download the old version of the nuget from http://www.siteextensions.net/api/v2/package/letsencrypt/0.5.9
The the nupkg is a zip file, that you can extract, and upload the contents from the content folder to D:\home\siteextension\letsencrypt Then you probably need to restart the kudu site, but then you should be running the old version.
@sjkp. that worked for me. I had this issue, my site has not had a valid cert for a while now :-(
I had to re-enter the application settings again, restarted and all I s well again until next time :-)
