azure-docs icon indicating copy to clipboard operation
azure-docs copied to clipboard

Azure function app caching issue with key vault reference

Open TestUser0811 opened this issue 4 years ago • 27 comments

My Application settings of Azure function app, has keyvault reference like @Microsoft.KeyVault(SecretUri=https://myvalut.vault.azure.net/secrets/mySecret/), whenever new version of "mySecret" introduced in keyvault, Azure function still renders the old/stale value not the latest value/version of "mySecret".

If I open up app setting using KUDU (https://myhttpfunc.scm.azurewebsites.net/api/settings) I can see "mySecret":"onemoretry" where "onemoretry" is the old value. Only Azure func app restart updates "mySecret" with new value.

How to make Azure function which uses keyvault reference to fetch latest value/version of the targeted keyvault secret as & when it get updated, without Azure function restart?

Can you please clarify whether the value returned would be cached on the Azure function app or if it would incur a request to KeyVault every time that App Setting was queried by the code?

If function app needs to be restarted everytime after keyvault secret vault update then what is the purpose of the below keyvault reference support without version number? @Microsoft.KeyVault(SecretUri=https://myvalut.vault.azure.net/secrets/mySecret/),


Document Details

Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

TestUser0811 avatar Apr 04 '20 13:04 TestUser0811

Some quick update would be appreciated.

TestUser0811 avatar Apr 04 '20 15:04 TestUser0811

The scope for these doc issues is to help identify gaps and issues with the documentation. Is there something in any specific Azure doc you are looking to have updated or are having issues with? If yes, please provide us with the link to the documentation you are referring to so we can better assist.

Else, if you are looking for help with debugging issues I would suggest reaching out to the MSDN and Stack Overflow communities. Our team will help you there.

Karishma-Tiwari-MSFT avatar Apr 04 '20 17:04 Karishma-Tiwari-MSFT

It is not about the document, it is more about the functionality of App services as I have mentioned in my original post. Can you move this concern or tag right team to answer/clarify the query?

TestUser0811 avatar Apr 05 '20 13:04 TestUser0811

@TestUser0811 I believe this is the current intended behavior. The secret is fetched when the function first warms up and if the function were to scale down to zero, it will pick the latest secret at the time of warming up again.

@mattchenderson Could you please confirm and share more insights here?

We are assigning this issue to the content author for further review and to update the docs accordingly.

PramodValavala-MSFT avatar Apr 06 '20 10:04 PramodValavala-MSFT

@PramodValavala-MSFT @mattchenderson,
"The secret is fetched when the function first warms up and if the function were to scale down to zero" this statement of yours is not true.

Only function app restart is fetching the latest secret & it is painful to restart function app every time you update the keyvault secret version & it is definite caveat Microsoft has to patch out.

Even tried out updating the function app's application settings with the newly version of keyvault secret by using the REST API & the application settings value get updated as expected but still function is rendering the previous value of the keyvault secret not the latest value. Function App considering the latest version of the secret only after its restart.

Help me with the possible solution where function can able to pick up the latest value of key vault secret version with out Function App restart.

Format of Keyvault reference configured Function App's application setting, @Microsoft.KeyVault(SecretUri=https://myvalut.vault.azure.net/secrets/mySecret/),

TestUser0811 avatar Apr 07 '20 16:04 TestUser0811

some update would be appreciated.

TestUser0811 avatar Apr 10 '20 13:04 TestUser0811

Does restarting the function app pick up the latest value? It didn't seem to work for me. What I had to do is to make a minor update to the url. Actually removed the "/" at the end and saved it. Then it picked up the latest.

kmnalluri avatar Mar 12 '21 14:03 kmnalluri

We are facing this as well. Tried restarting the function app, removed the "/" from key vault url, but the function app picks up the old secret value.

avanti-ac avatar May 06 '21 17:05 avanti-ac

This really needs to be fixed - a restart should clear any cache being used!

johnmc-tc avatar Jun 04 '21 14:06 johnmc-tc

Use the "SecretVersion" in your configuration reference. @Microsoft.KeyVault(VaultName=myvault;SecretName=mysecret;SecretVersion=ec96f02080254f109c51a1f14cdb1931) Use Key Vault reference

kkurkowski avatar Jun 16 '21 17:06 kkurkowski

Use the "SecretVersion" in your configuration reference. @Microsoft.KeyVault(VaultName=myvault;SecretName=mysecret;SecretVersion=ec96f02080254f109c51a1f14cdb1931) Use Key Vault reference

That would mean updating configuration when keyvault value has a new version added, which is not viable or acceptable in most cases.

johnmc-tc avatar Jun 16 '21 17:06 johnmc-tc

Is there any update on this obvious problem? It really needs to be resolved - restarting and stopping and then starting the function app has no effect on the function app seeing the new KV value.

johnmc-tc avatar Jul 30 '21 09:07 johnmc-tc

Restarting my FA didn't pick up the latest KV value :(

dhardie avatar Aug 11 '21 12:08 dhardie

I am also facing the same issue. Was there any resolution around this?

pbhakuni avatar Sep 03 '21 13:09 pbhakuni

Same issue. Refreshing the app-service doesn't fetch newest value. I need to make a change in the config of the app service and save that. It will ask for a refresh and that will trigger it.

xigolle avatar Sep 30 '21 14:09 xigolle

@PramodValavala-MSFT @Karishma-Tiwari-MSFT @mattchenderson Also having the same issue with AppServices, restarting isn't enough to get the most recent secret value. I can even disable all older versions in my keyvault and it still uses the old version. Because of this I'm fairly certain that this is a caching issue.

After a certain time and restarting it just works. I guess this happens when the cache TTL is finally reached.

stefan-schweiger avatar Nov 26 '21 17:11 stefan-schweiger

Any update on this issue? Function restart is not picking the latest version of the KeyVault secrets, only configuration change refreshes the latest version of the KeyVault secrets. Its high maintenance to continue with workaround if secret is referenced in multiple functions/app services.

vechkar avatar Jan 04 '22 21:01 vechkar

My solution for now is to specify ;SecretVersion=... in the config, but this is a major pain point as you need to update this everytime you update your secret

stefan-schweiger avatar Jan 05 '22 08:01 stefan-schweiger

I have the same issue. This "cache" persists even through multiple app restarts and code redeployments. Using a SecretVersion is not a solution but a workaround and it means I have to write even more automation to handle this.

KostovBG avatar May 23 '22 09:05 KostovBG

same here. in our case, the cache can live up to 5 hours and even we restart the function app and it still cannot flush the cache.

AlphaWong avatar Jul 07 '22 05:07 AlphaWong

This madness caused us a 6-hour P1 recently.

m3talstorm avatar Jul 13 '22 10:07 m3talstorm

This is burning me too and I say that this is nothing more than pure poop! MSFT, can you please fix this?

phazlett avatar Sep 13 '22 21:09 phazlett

I am also suffering on this. Function (v4 isolated) is still seeing the old value even though the function is restarted. Pls fix this!

AaliAlikoski avatar Sep 14 '22 13:09 AaliAlikoski

We just had the same issue with Function App (2 instances P1V3). Changing KeyVault reference using SecretVersion and roll it back without SecretVersion works for our issue.

alexander-lanin avatar Sep 21 '22 21:09 alexander-lanin

Same here with Function App v4 (P1V3). Looking forward on getting this fixed to save hours of troubleshooting in the future 😅

tlaukkanen avatar Dec 02 '22 13:12 tlaukkanen

+1 for this issue to be fixed - should pick up latest version by default

tonitakala avatar Dec 07 '22 11:12 tonitakala

+1

idusortus avatar Dec 12 '22 20:12 idusortus

We would also appreciate a fix for this. A restart should not use the cached value

plamber avatar Dec 29 '22 06:12 plamber

Same problem here. Please fix it.

twilson-gl avatar Jan 09 '23 17:01 twilson-gl

Adding on to the pile, here. At minimum, a function app restart or code redeployment should cause a cache refresh. With a properly set up CI/CD, it's a mere button click to restart/redeploy to pick up new value, and I don't mind that.

bcline760 avatar Jan 10 '23 21:01 bcline760