Azure
[Bug] Worker failed to index functions
Expected Behavior
I expected the worker to start without problems
Actual Behavior
After deployment I could see an error in Application Insights. The python worker seems to be unable to start.
Steps to Reproduce
My test azure function works using the following: Runtime version: 4.1044.300.1 Python 3.12 opentelemetry-api in requirements.txt
My production azure function works using the following: Runtime version: 4.1045.100.0 Python 3.12 opentelemetry-api==1.24.0 in requirements.txt
Whenever I change opentelemetry-api==1.24.0 to opentelemetry-api I receive the error below.
Relevant code being tried
Relevant log output
Result: Failure
Type:
Exception: StopIteration:
Stack: File "/azure-functions-host/workers/python/3.12/LINUX/X64/azure_functions_worker/dispatcher.py", line 485, in _handle__functions_metadata_request
self.load_function_metadata(
File "/azure-functions-host/workers/python/3.12/LINUX/X64/azure_functions_worker/dispatcher.py", line 465, in load_function_metadata
self.index_functions(function_path, function_app_directory)) \
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/azure-functions-host/workers/python/3.12/LINUX/X64/azure_functions_worker/dispatcher.py", line 847, in index_functions
indexed_functions = loader.index_function_app(function_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/azure-functions-host/workers/python/3.12/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/azure-functions-host/workers/python/3.12/LINUX/X64/azure_functions_worker/loader.py", line 244, in index_function_app
imported_module = importlib.import_module(module_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/3/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 999, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/home/site/wwwroot/function_app.py", line 17, in <module>
from azure.monitor.opentelemetry import configure_azure_monitor
File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/monitor/opentelemetry/__init__.py", line 7, in <module>
from azure.monitor.opentelemetry._configure import configure_azure_monitor
File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/monitor/opentelemetry/_configure.py", line 10, in <module>
from opentelemetry.instrumentation.instrumentor import ( # type: ignore
File "/home/site/wwwroot/.python_packages/lib/site-packages/opentelemetry/instrumentation/instrumentor.py", line 26, in <module>
from opentelemetry.instrumentation._semconv import (
File "/home/site/wwwroot/.python_packages/lib/site-packages/opentelemetry/instrumentation/_semconv.py", line 19, in <module>
from opentelemetry.instrumentation.utils import http_status_to_status_code
File "/home/site/wwwroot/.python_packages/lib/site-packages/opentelemetry/instrumentation/utils.py", line 25, in <module>
from opentelemetry import context, trace
File "/home/site/wwwroot/.python_packages/lib/site-packages/opentelemetry/context/__init__.py", line 70, in <module>
_RUNTIME_CONTEXT = _load_runtime_context()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/site/wwwroot/.python_packages/lib/site-packages/opentelemetry/context/__init__.py", line 60, in _load_runtime_context
return next( # type: ignore
^^^^^^^^^^^^^^^^^^^^^
requirements.txt file
# Azure Functions
azure-functions
azure-storage-queue
azure-storage-blob
# Authentication
PyJWT[crypto]
requests
# Connections
aiohttp
# JSON
ujson
# Models
pydantic[email]
country-converter
# Snowflake
snowflake-connector-python>=3.7.0,<4.0.0
snowflake-sqlalchemy
# OpenTelemetry and Application Insights
azure-monitor-opentelemetry
opentelemetry-instrumentation-logging
opentelemetry-api
Where are you facing this problem?
Production Environment (explain below)
Function app name
No response
Additional Information
No response
Hi @o-lovgren, thanks for reporting.
I wasn't able to repro this on a Python 3.12 Flex app using runtime version 4.1045 and the same requirements. For reference, I'll list the specific dependency versions installed below. The latest opentelemetry-api version is 1.39.0 though, and 1.24.0 is over a year older - it looks like there may be a conflict somewhere else in the dependency chain that's causing the imports to fail.
PyJWT 2.10.1
aiohappyeyeballs 2.6.1
aiohttp 3.13.2
aiosignal 1.4.0
annotated-types 0.7.0
asgiref 3.11.0
asn1crypto 1.5.1
attrs 25.4.0
azure-core 1.36.0
azure-core-tracing-opentelemetry 1.0.0b12
azure-functions 1.24.0
azure-identity 1.25.1
azure-monitor-opentelemetry 1.8.3
azure-monitor-opentelemetry-exporter 1.0.0b46
azure-storage-blob 12.27.1
azure-storage-queue 12.14.1
boto3 1.42.6
botocore 1.42.6
certifi 2025.11.12
cffi 1.17.1
charset_normalizer 3.4.4
country-converter 1.3.2
cryptography 46.0.0
dnspython 2.8.0
email-validator 2.3.0
filelock 3.20.0
frozenlist 1.8.0
greenlet 3.3.0
idna 3.11
importlib-metadata 8.7.0
isodate 0.7.2
jmespath 1.0.1
markupsafe 3.0.3
msal 1.34.0
msal-extensions 1.3.1
msrest 0.7.1
multidict 6.7.0
numpy 2.3.5
oauthlib 3.3.1
opentelemetry-api 1.39.0
opentelemetry-instrumentation 0.60b0
opentelemetry-instrumentation-asgi 0.60b0
opentelemetry-instrumentation-dbapi 0.60b0
opentelemetry-instrumentation-django 0.60b0
opentelemetry-instrumentation-fastapi 0.60b0
opentelemetry-instrumentation-flask 0.60b0
opentelemetry-instrumentation-logging 0.60b0
opentelemetry-instrumentation-psycopg2 0.60b0
opentelemetry-instrumentation-requests 0.60b0
opentelemetry-instrumentation-urllib 0.60b0
opentelemetry-instrumentation-urllib3 0.60b0
opentelemetry-instrumentation-wsgi 0.60b0
opentelemetry-resource-detector-azure 0.1.5
opentelemetry-sdk 1.39.0
opentelemetry-semantic-conventions 0.60b0
opentelemetry-util-http 0.60b0
packaging 25.0
pandas 2.3.3
platformdirs 4.5.1
propcache 0.4.1
psutil 7.1.3
pyOpenSSL 25.3.0
pycparser 2.23
pydantic 2.12.5
pydantic-core 2.41.5
python-dateutil 2.9.0.post0
pytz 2025.2
requests 2.32.5
requests-oauthlib 2.0.0
s3transfer 0.16.0
six 1.17.0
snowflake-connector-python 3.18.0
snowflake-sqlalchemy 1.8.2
sortedcontainers 2.4.0
sqlalchemy 2.0.45
tomlkit 0.13.3
typing-extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.2
ujson 5.11.0
urllib3 2.6.1
werkzeug 3.1.4
wrapt 1.17.3
yarl 1.22.0
zipp 3.23.0
Hi @hallvictoria I don't know if it makes a difference, but we are running Premium v3 as the hosting plan.
I can also confirm that we are using the exact same dependency list when using opentelemetry-api in requirements.txt. This was taken from my pipeline output:
PyJWT-2.10.1 aiohappyeyeballs-2.6.1 aiohttp-3.13.2 aiosignal-1.4.0 annotated-types-0.7.0 asgiref-3.11.0 asn1crypto-1.5.1 attrs-25.4.0 azure-core-1.36.0 azure-core-tracing-opentelemetry-1.0.0b12 azure-functions-1.24.0 azure-identity-1.25.1 azure-monitor-opentelemetry-1.8.3 azure-monitor-opentelemetry-exporter-1.0.0b46 azure-storage-blob-12.27.1 azure-storage-queue-12.14.1 boto3-1.42.7 botocore-1.42.7 certifi-2025.11.12 cffi-1.17.1 charset_normalizer-3.4.4 country-converter-1.3.2 cryptography-46.0.0 dnspython-2.8.0 email-validator-2.3.0 filelock-3.20.0 frozenlist-1.8.0 greenlet-3.3.0 idna-3.11 importlib-metadata-8.7.0 isodate-0.7.2 jmespath-1.0.1 markupsafe-3.0.3 msal-1.34.0 msal-extensions-1.3.1 msrest-0.7.1 multidict-6.7.0 numpy-2.3.5 oauthlib-3.3.1 opentelemetry-api-1.39.0 opentelemetry-instrumentation-0.60b0 opentelemetry-instrumentation-asgi-0.60b0 opentelemetry-instrumentation-dbapi-0.60b0 opentelemetry-instrumentation-django-0.60b0 opentelemetry-instrumentation-fastapi-0.60b0 opentelemetry-instrumentation-flask-0.60b0 opentelemetry-instrumentation-logging 0.60b0 opentelemetry-instrumentation-psycopg2-0.60b0 opentelemetry-instrumentation-requests-0.60b0 opentelemetry-instrumentation-urllib-0.60b0 opentelemetry-instrumentation-urllib3-0.60b0 opentelemetry-instrumentation-wsgi-0.60b0 opentelemetry-resource-detector-azure-0.1.5 opentelemetry-sdk-1.39.0 opentelemetry-semantic-conventions-0.60b0 opentelemetry-util-http-0.60b0 packaging-25.0 pandas-2.3.3 platformdirs-4.5.1 propcache-0.4.1 psutil-7.1.3 pyOpenSSL-25.3.0 pycparser-2.23 pydantic-2.12.5 pydantic-core-2.41.5 python-dateutil-2.9.0.post0 pytz-2025.2 requests-2.32.5 requests-oauthlib-2.0.0 s3transfer-0.16.0 six-1.17.0 snowflake-connector-python-3.18.0 snowflake-sqlalchemy-1.8.2 sortedcontainers-2.4.0 sqlalchemy-2.0.45 tomlkit-0.13.3 typing-inspection-0.4.2 typing_extensions-4.15.0 tzdata-2025.2 ujson-5.11.0 urllib3-2.6.1 werkzeug-3.1.4 wrapt-1.17.3 yarl-1.22.0 zipp-3.23.0
Today I also tried pinning all my relevant opentelemetry packages as stated here: https://github.com/Azure/azure-functions-python-worker/issues/1808#issuecomment-3612223279
I used the following in my requirements.txt: azure-monitor-opentelemetry==1.8.2 opentelemetry-instrumentation-logging==0.59b0 opentelemetry-api==1.38.0
I can see that these were the packages used in my previous successful deployment. I verified the packages in my pipeline logs, but I am still gettting the same error. The only difference I can find is that I am running Premium instead of Flex.
Worth noting is that all builds work when I have tried locally on Function runtime: 4.1044.400.25520. All builds also work on my test azure function with Function runtime: 4.1044.300.1
I tried creating a 3.12 app on Premium (function runtime version 4.1045) with those dependencies & import statement, but I still wasn't able to reproduce the issue.
How are you deploying your app? I used Core Tools - func azure functionapp publish.
There weren't any Python worker updates from 4.1044 to 4.1045, so it doesn't seem likely that the host update version is the cause. If you're willing to share your app name privately, we can also take a look from the backend to see if there are any errors.
I tried creating a 3.12 app on Premium (function runtime version 4.1045) with those dependencies & import statement, but I still wasn't able to reproduce the issue.
How are you deploying your app? I used Core Tools -
func azure functionapp publish.There weren't any Python worker updates from 4.1044 to 4.1045, so it doesn't seem likely that the host update version is the cause. If you're willing to share your app name privately, we can also take a look from the backend to see if there are any errors.
I am using https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/azure-function-app-v1?view=azure-pipelines for deployment.
UTC execution time: Mon, 15 Dec 2025 07:13:13 GUID: bddf70e0-6712-4b96-b5d6-1772a78452c3 Region: Norway East
Is 404 the expected return status from https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately ?
Also, these are the import statements that I am using:
from azure.monitor.opentelemetry import configure_azure_monitor from opentelemetry import trace from opentelemetry.propagate import extract from opentelemetry.trace import SpanKind from opentelemetry.instrumentation.logging import LoggingInstrumentor
Is 404 the expected return status
Yep, thanks for that.
Two things to be called out here:
- Remote build is not being used - local builds are okay but require your local / pipeline environment to match what is in production. Your app is running on Python 3.12, so you'll need to have an ubuntu image and use Python 3.12 to build the app (example).
- The exact exception is being thrown on this line - it's failing to load the context.
Could you try deploying your app using remote build? You can use Core Tools or the VSCode extension if you don't want to make any changes to your pipeline or deploy to a fresh app using remote build to narrow down if the issue is with the deployment pipeline.
Is 404 the expected return status
Yep, thanks for that.
Two things to be called out here:
- Remote build is not being used - local builds are okay but require your local / pipeline environment to match what is in production. Your app is running on Python 3.12, so you'll need to have an ubuntu image and use Python 3.12 to build the app (example).
- The exact exception is being thrown on this line - it's failing to load the context.
Could you try deploying your app using remote build? You can use Core Tools or the VSCode extension if you don't want to make any changes to your pipeline or deploy to a fresh app using remote build to narrow down if the issue is with the deployment pipeline.
Thank you so much! After applying the given application settings for Remote Build, everything is working as expected. It would be interesting to know why though. I can see that Oryx creates a build with the exact same dependencies as the ones I have used all along ...
Glad that helped! We recommend using remote build in general to ensure that compatible wheels are installed to match the production environment.
I'm not too familiar with the opentelemetry packages & their dependency hierarchy, but there are a couple of other packages in your requirements that have platform specific builds - for example, cryptography and cffi. When not using remote build, the local environment must match the production environment to ensure that the correct dependency wheels are installed.
It looks like prior to remote build, an incompatible wheel was installed based on the local environment. When using remote build, even though the same version was installed, the correct wheel was pulled to match the production environment. The previous opentelemetry-api version was fairly old, so they may have had an update in their dependencies that resulted in this.