Protobuf v4 error - Descriptors cannot be created directly.

[YunchuWang]

Announcement: Azure python function apps pinning protobuf package to v4.x.x directly or indirectly (ex. use another library which requires protobuf v4) malfunction as protobuf v4.x.x introduces breaking changes

Problem: There is breaking change introduced in new protobuf package v4.x.x version, ref at https://github.com/protocolbuffers/protobuf/issues/10064

Azure python function worker currently using protobuf package of v3.x.x version. Since protobuf v4 is released, it is observed many python function apps which pin protobuf to v4.x.x in the requirement.txt are affected. Detailed error logs are as below. Caution, if your function app is affected, please refer to mitigation step section below!!!

 [Information] File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/protos/shared/NullableTypes_pb2.py", line 38, in <module>
 [Information] _descriptor.FieldDescriptor(
 [Information] File "/home/site/wwwroot/.python_packages/lib/site-packages/google/protobuf/descriptor.py", line 560, in __new__
 [Information] _message.Message._CheckCalledFromGeneratedFile()
 [Error] TypeError: Descriptors cannot not be created directly.
 [Information] If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
 [Information] If you cannot immediately regenerate your protos, some other possible workarounds are:
 [Information] 1. Downgrade the protobuf package to 3.20.x or lower.
 [Information] 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
 [Information] More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

Mitigation Two ways to mitigate the issue:

1. Set PYTHON_ISOLATE_WORKER_DEPENDENCIES in AppSetting to 1. (For reference, please check https://docs.microsoft.com/en-us/azure/azure-functions/functions-app-settings#python_isolate_worker_dependencies-preview)
2. For users not requiring protobuf v4.x.x versions in your function app requirement.txt, please pin protobuf >= 3.19.3, == 3.*

[GinSiuCheng]

• This issued started showing up in the past week for Python Azure Functions with premium (EP) plans (Regions: USE2, USW2, USE, USC) causing function host to be dead in the water/stalled.
• Functions written did not explicitly require/use protobuf, think the root cause is that language workers required it as noted here: https://github.com/Azure/azure-functions-language-worker-protobuf
• Recent deployments / new code changes and app restarts seems to trigger this. This points to the underlying container image upgrading from 3.20.X to 4.21 causing breaking changes in customer environments.
• The above mitigation is a temporary fix and brought things to steady state, but breaking changes like this should be further tested before roll out to end users.

[ripsj]

For us this issue was detected on 22/06/2022, and problem is there are no logs on Function Apps. The one helpful clue we found for this was when looking at the container logs, it showed "unhealthy, Stopping site" Temporal solution has worked so far, hopefully Microsoft fixes this soon, and let the community know, it took us a LOT of hours to be able to fix this issue.

[gerardsimons]

Thanks for posting this, it is indeed a terrible error since it only started showing up for me when switching to a Premium function and looking at the logs in the advanced tools ... This is an awful awful bug that has caused my company a lot of time and effort, please fix!

[gavin-aguiar]

@gerardsimons we have fixed this bug and is currently being rolled out. Runtime version 4.9.1 will have the fix for this.

[waltherg]

How do I get a hold of one of these upgraded runtimes with the fix? Redeploying an app today on an EP plan in West Europe gave me this. Multiple app restarts did not help.

[henrikhansen428]

How do I get a hold of one of these upgraded runtimes with the fix? Redeploying an app today on an EP plan in West Europe gave me this. Multiple app restarts did not help.

@waltherg Have you tried any of the mitagation options mentioned in the issue description? Setting PYTHON_ISOLATE_WORKER_DEPENDENCIES to 1 in AppSettings worked for me

[waltherg]

How do I get a hold of one of these upgraded runtimes with the fix? Redeploying an app today on an EP plan in West Europe gave me this. Multiple app restarts did not help.

@waltherg Have you tried any of the mitagation options mentioned in the issue description? Setting PYTHON_ISOLATE_WORKER_DEPENDENCIES to 1 in AppSettings worked for me

Good call. Yes setting that environment variable did help, thanks. I would've just hoped all runners had been upgraded by now. As others have commented this issue is extremely time consuming to work out because the App Insights exception logs aren't descriptive at all.