Cannot deploy to a function hosted on Linux Consumption plan when AzureWebJobsStorage config setting is Key Vault reference

[rjygraham]

When a function hosted on Linux Consumption plan has AzureWebJobsStorage configuration setting set to valid Key Vault reference, errors block successful deployment via Az CLI and Functions CLI.

Investigative information

Please provide the following:

• Timestamp: 2021-03-26 04:06:00 UTC
• Function App version: ~3 runtime, dotnet-isolated (.NET 5)
• Function App name: oe-twitch-eus-identity-2021-03-15-func
• Function name(s) (as appropriate): N/A
• Invocation ID: N/A
• Region: East US

Repro steps

1. Create Function App using Linux Consumption plan
2. Enable Function System-Assigned Managed Identity
3. Create Azure Key Vault and grant Function App identity Secret List/Get permissions in Key Vault ACL
4. Add Key Vault secret containing the Function's Storage Account connection string as a value
5. Update the Function App AzureWebJobsStorage to be a Key Vault reference
6. Attempt to deploy a function app via Az CLI or Functions CLI.

Expected behavior

• Step 6 will successfully deploy the code to the Function App

Actual behavior

Az CLI fails with the following errors:

az functionapp deployment source config-zip -g OE-TWITCH-EUS -n oe-twitch-eus-identity-2021-03-15-func --src .\bin\publish\publish.zip
The command failed with an unexpected error. Here is the traceback:
You need to provide an account name and either an account_key or sas_token when creating a storage service.
Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-taryuzhh\knack\cli.py", line 233, in invoke
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\commands\__init__.py", line 660, in execute
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\commands\__init__.py", line 723, in _run_jobs_serially
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\commands\__init__.py", line 716, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-l7ikczg1\six.py", line 703, in reraise
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\commands\__init__.py", line 694, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\commands\__init__.py", line 331, in __call__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\__init__.py", line 808, in default_command_handler
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-ibh5mnup\azure\cli\command_modules\appservice\custom.py", line 385, in enable_zip_deploy_functionapp
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-ibh5mnup\azure\cli\command_modules\appservice\custom.py", line 543, in upload_zip_to_storage
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-r6yjb8kq\azure\multiapi\storage\v2018_11_09\blob\blockblobservice.py", line 146, in __init__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-r6yjb8kq\azure\multiapi\storage\v2018_11_09\blob\baseblobservice.py", line 209, in __init__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-r6yjb8kq\azure\multiapi\storage\v2018_11_09\common\_connection.py", line 110, in get_service_parameters
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-r6yjb8kq\azure\multiapi\storage\v2018_11_09\common\_connection.py", line 161, in _from_connection_string
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-r6yjb8kq\azure\multiapi\storage\v2018_11_09\common\_connection.py", line 85, in __init__
ValueError: You need to provide an account name and either an account_key or sas_token when creating a storage service.
To open an issue, please run: 'az feedback'

Functions CLI fails with the following errors:

func azure functionapp publish oe-twitch-eus-identity-2021-03-15-func
Microsoft (R) Build Engine version 16.9.0+57a23d249 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
  func-dotnet5 -> C:\Users\ryan\source\repos\func-dotnet5\bin\publish\func-dotnet5.dll
  Determining projects to restore...
  Restored C:\Users\ryan\AppData\Local\Temp\ure1mi1o.wlg\WorkerExtensions.csproj (in 1.14 sec).
  WorkerExtensions -> C:\Users\ryan\AppData\Local\Temp\ure1mi1o.wlg\buildout\Microsoft.Azure.Functions.Worker.Extensions.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:08.79


Getting site publishing info...
Uploading package...
Error creating a Blob container reference. Please make sure your connection string in "AzureWebJobsStorage" is valid
Retry: 1 of 3
Error creating a Blob container reference. Please make sure your connection string in "AzureWebJobsStorage" is valid
Retry: 2 of 3
Error creating a Blob container reference. Please make sure your connection string in "AzureWebJobsStorage" is valid
Retry: 3 of 3
Error creating a Blob container reference. Please make sure your connection string in "AzureWebJobsStorage" is valid

Known workarounds

• Do not use a Key Vault reference for AzureWebJobsStorage setting on a Linux Consumption plan
• Use a Windows Consumption plan appears to work with AzureWebJobsStorage as Key Vault reference

Related information

Provide any related information

• Programming language used: .NET 5
• Links to source: N/A
• Bindings used: N/A

[v-bbalaiagar]

Hi @balag0, Could you please look into this issue. Transferring this issue to core tools for further investigation.

[balag0]

Thanks. yes. this is not supported in core tools currently. will take care of this. assigned this to myself cc @pragnagopa

[pragnagopa]

@balag0 - Assigned this current sprint. Let me know if this needs to be assigned to next sprint instead.

[fabiocav]

Assigning this to sprint 101

[balag0]

Yes, didn't get a chance to work on this. Sprint 101 is good. Thanks

[fabiocav]

@balag0 moving this back to triaged, please feel free to assign to a sprint when we're pretty confident we'll be able to close this. Thanks!

[balag0]

I am working on this, but this needs some other pieces implemented first in kudu and elsewhere which I am focusing on first.

[ChrisProlls]

Any news on this ?

[cveld]

Just tested, Func cli still fails. Error message: Error creating a Blob container reference. Please make sure your connection string in "AzureWebJobsStorage" is valid

Interestingly deploying from visual studio 2022 (17.2) works fine.

Related, the Azure devops pipeline task fails. Unbelievable I thought I was doing something wrong here. I was also misled by the message in Portal:

The portal is not able to confirm the status of your Key Vault reference at this time. Please confirm the status directly from the app by checking whether the environment variables have resolved.

Just opened a ticket: https://github.com/microsoft/azure-pipelines-tasks/issues/16749