azure-search-openai-demo icon indicating copy to clipboard operation
azure-search-openai-demo copied to clipboard

Published 20 hours ago • verified publisher icon Azure-Samples

ModuleNotFoundError: No module named 'main' opentelemetry dep issues

Open cforce opened this issue 1 year ago • 11 comments

Followed all debug steps of https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/appservice.md until i found an unexplainable issue ats startup_

2024/07/27 11:24:47 main.go:50: Retry: response 200 2024/07/27 11:24:47 main.go:50: Retry: exit due to non-retriable status code (✓) Done: Deploying service backend

  • Endpoint: https://XXXXXXXXXXXXXXXX.azurewebsites.net/

SUCCESS: Your application was deployed to Azure in 30 minutes 22 seconds. You can view the resources created under the resource group ece-prodweub-docbot-rg in Azure Portal: 1https://portal.azure.com/#@/resource/subscriptions/YYYYYYYYYYYYYYYYYYYYYYYYYYYY/resourceGroups/RRRRRRRRRRRRR-rg/overview

2024-07-27T00:27:38.7892647Z _____
2024-07-27T00:27:38.7894244Z / _ \ __________ _________ ____
2024-07-27T00:27:38.7894357Z / /\ \__ / | _ __ _/ __ \ 2024-07-27T00:27:38.7894406Z / | / /| | /| | /\ / 2024-07-27T00:27:38.7894455Z _|__ /_____ _/ || __ > 2024-07-27T00:27:38.7894502Z / / / 2024-07-27T00:27:38.7894548Z A P P S E R V I C E O N L I N U X 2024-07-27T00:27:38.7894592Z 2024-07-27T00:27:38.7894663Z Documentation: http://aka.ms/webapp-linux 2024-07-27T00:27:38.7894710Z Python 3.11.8 2024-07-27T00:27:38.7894756Z Note: Any data outside '/home' is not persisted 2024-07-27T00:27:40.2836783Z Starting OpenBSD Secure Shell server: sshd. 2024-07-27T00:27:40.3314078Z WEBSITES_INCLUDE_CLOUD_CERTS is not set to true. 2024-07-27T00:27:40.6308099Z Site's appCommandLine: python3 -m gunicorn main:app 2024-07-27T00:27:40.9948027Z Starting periodic command scheduler: cron. 2024-07-27T00:27:41.0053470Z Launching oryx with: create-script -appPath /home/site/wwwroot -output /opt/startup/startup.sh -virtualEnvName antenv -defaultApp /opt/defaultsite -userStartupCommand 'python3 -m gunicorn main:app' 2024-07-27T00:27:41.1039522Z Could not find build manifest file at '/home/site/wwwroot/oryx-manifest.toml' 2024-07-27T00:27:41.1057743Z Could not find operation ID in manifest. Generating an operation id... 2024-07-27T00:27:41.1233128Z Build Operation ID: a665f202-54d0-443c-944e-11efc28bac7b 2024-07-27T00:27:41.6327554Z Oryx Version: 0.2.20240619.2, Commit: cf006407a02b225f59dccd677986973c7889aa50, ReleaseTagName: 20240619.2 2024-07-27T00:27:41.6543031Z Writing output script to '/opt/startup/startup.sh' 2024-07-27T00:27:41.7411267Z WARNING: Could not find virtual environment directory /home/site/wwwroot/antenv. 2024-07-27T00:27:41.7428213Z WARNING: Could not find package directory /home/site/wwwroot/oryx_packages. 2024-07-27T00:27:43.5036823Z [2024-07-27 00:27:43 +0000] [81] [INFO] Starting gunicorn 22.0.0 2024-07-27T00:27:43.5052472Z [2024-07-27 00:27:43 +0000] [81] [INFO] Listening at: http://0.0.0.0:8000 (81) 2024-07-27T00:27:43.5214209Z [2024-07-27 00:27:43 +0000] [81] [INFO] Using worker: sync 2024-07-27T00:27:43.5345590Z [2024-07-27 00:27:43 +0000] [82] [INFO] Booting worker with pid: 82 2024-07-27T00:27:43.5931697Z [2024-07-27 00:27:43 +0000] [82] [ERROR] Exception in worker process 2024-07-27T00:27:43.5932888Z Traceback (most recent call last): 2024-07-27T00:27:43.5932948Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker 2024-07-27T00:27:43.5933001Z worker.init_process() 2024-07-27T00:27:43.5933058Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process 2024-07-27T00:27:43.5933174Z self.load_wsgi() 2024-07-27T00:27:43.5933225Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi 2024-07-27T00:27:43.5933270Z self.wsgi = self.app.wsgi() 2024-07-27T00:27:43.5933315Z ^^^^^^^^^^^^^^^ 2024-07-27T00:27:43.5933363Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi 2024-07-27T00:27:43.5933429Z self.callable = self.load() 2024-07-27T00:27:43.5933474Z ^^^^^^^^^^^ 2024-07-27T00:27:43.5933522Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load 2024-07-27T00:27:43.5933568Z return self.load_wsgiapp() 2024-07-27T00:27:43.5933614Z ^^^^^^^^^^^^^^^^^^^ 2024-07-27T00:27:43.5933678Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp 2024-07-27T00:27:43.5933727Z return util.import_app(self.app_uri) 2024-07-27T00:27:43.5933773Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-07-27T00:27:43.5933822Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/util.py", line 371, in import_app 2024-07-27T00:27:43.5933867Z mod = importlib.import_module(module) 2024-07-27T00:27:43.5933913Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-07-27T00:27:43.5933977Z File "/opt/python/3/lib/python3.11/importlib/init.py", line 126, in import_module 2024-07-27T00:27:43.5934025Z return bootstrap.gcd_import(name[level:], package, level) 2024-07-27T00:27:43.5934073Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-07-27T00:27:43.5934120Z File "", line 1204, in gcd_import 2024-07-27T00:27:43.5934183Z File "", line 1176, in find_and_load 2024-07-27T00:27:43.5934232Z File "", line 1140, in find_and_load_unlocked 2024-07-27T00:27:43.5934278Z ModuleNotFoundError: No module named 'main' 2024-07-27T00:27:43.6121435Z [2024-07-27 00:27:43 +0000] [82] [INFO] Worker exiting (pid: 82) 2024-07-27T00:27:43.8065597Z [2024-07-27 00:27:43 +0000] [81] [ERROR] Worker (pid:82) exited with code 3 2024-07-27T00:27:43.8224651Z [2024-07-27 00:27:43 +0000] [81] [ERROR] Shutting down: Master 2024-07-27T00:27:43.8225005Z [2024-07-27 00:27:43 +0000] [81] [ERROR] Reason: Worker failed to boot. 2024-07-27T00:28:25.530406476Z _____
2024-07-27T00:28:25.530463977Z / _ \ __________ _________ ____
2024-07-27T00:28:25.530470977Z / /\ \ / | _ __ _/ __ \ 2024-07-27T00:28:25.530475877Z / | / /| | /| | /\ / 2024-07-27T00:28:25.530480077Z _|_ /_____ _/ || __ > 2024-07-27T00:28:25.530484477Z / / / 2024-07-27T00:28:25.530488578Z A P P S E R V I C E O N L I N U X 2024-07-27T00:28:25.530492778Z 2024-07-27T00:28:25.530496678Z Documentation: http://aka.ms/webapp-linux 2024-07-27T00:28:25.530500678Z Python 3.11.8 2024-07-27T00:28:25.530504578Z Note: Any data outside '/home' is not persisted 2024-07-27T00:28:30.024985566Z Starting OpenBSD Secure Shell server: sshd. 2024-07-27T00:28:30.154095574Z WEBSITES_INCLUDE_CLOUD_CERTS is not set to true. 2024-07-27T00:28:30.310384842Z Site's appCommandLine: python3 -m gunicorn main:app 2024-07-27T00:28:30.318372048Z Launching oryx with: create-script -appPath /home/site/wwwroot -output /opt/startup/startup.sh -virtualEnvName antenv -defaultApp /opt/defaultsite -userStartupCommand 'python3 -m gunicorn main:app' 2024-07-27T00:28:30.586568697Z Could not find build manifest file at '/home/site/wwwroot/oryx-manifest.toml' 2024-07-27T00:28:30.596054023Z Could not find operation ID in manifest. Generating an operation id... 2024-07-27T00:28:30.705904876Z Build Operation ID: 2e710c01-a499-4acf-95cb-17d9c74a10e5 2024-07-27T00:28:31.890939757Z Oryx Version: 0.2.20240501.1, Commit: f83f88d3cfb8bb6d3e2765e1dcd218eb0814a095, ReleaseTagName: 20240501.1 2024-07-27T00:28:31.930469980Z Writing output script to '/opt/startup/startup.sh' 2024-07-27T00:28:32.072753663Z WARNING: Could not find virtual environment directory /home/site/wwwroot/antenv. 2024-07-27T00:28:32.113394901Z WARNING: Could not find package directory /home/site/wwwroot/oryx_packages. 2024-07-27T00:28:34.328924018Z [2024-07-27 00:28:34 +0000] [63] [INFO] Starting gunicorn 22.0.0 2024-07-27T00:28:34.328957218Z [2024-07-27 00:28:34 +0000] [63] [INFO] Listening at: http://0.0.0.0:8000 (63) 2024-07-27T00:28:34.337424130Z [2024-07-27 00:28:34 +0000] [63] [INFO] Using worker: sync 2024-07-27T00:28:34.359122417Z [2024-07-27 00:28:34 +0000] [68] [INFO] Booting worker with pid: 68 2024-07-27T00:28:34.388854811Z [2024-07-27 00:28:34 +0000] [68] [ERROR] Exception in worker process 2024-07-27T00:28:34.388907811Z Traceback (most recent call last): 2024-07-27T00:28:34.388915511Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker 2024-07-27T00:28:34.388921212Z worker.init_process() 2024-07-27T00:28:34.388926112Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process 2024-07-27T00:28:34.388931312Z self.load_wsgi() 2024-07-27T00:28:34.388947512Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi 2024-07-27T00:28:34.388959712Z self.wsgi = self.app.wsgi() 2024-07-27T00:28:34.388964512Z ^^^^^^^^^^^^^^^ 2024-07-27T00:28:34.388969212Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi 2024-07-27T00:28:34.388973912Z self.callable = self.load() 2024-07-27T00:28:34.388978612Z ^^^^^^^^^^^ 2024-07-27T00:28:34.388983312Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load 2024-07-27T00:28:34.388988112Z return self.load_wsgiapp() 2024-07-27T00:28:34.388992413Z ^^^^^^^^^^^^^^^^^^^ 2024-07-27T00:28:34.388997213Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp 2024-07-27T00:28:34.389002013Z return util.import_app(self.app_uri) 2024-07-27T00:28:34.389006713Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-07-27T00:28:34.389011313Z File "/opt/python/3/lib/python3.11/site-packages/gunicorn/util.py", line 371, in import_app 2024-07-27T00:28:34.389016313Z mod = importlib.import_module(module) 2024-07-27T00:28:34.389020813Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-07-27T00:28:34.389025313Z File "/opt/python/3/lib/python3.11/importlib/init.py", line 126, in import_module 2024-07-27T00:28:34.389029913Z return _bootstrap._gcd_import(name[level:], package, level) 2024-07-27T00:28:34.389034613Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-07-27T00:28:34.389039213Z File "", line 1204, in _gcd_import 2024-07-27T00:28:34.389044513Z File "", line 1176, in _find_and_load 2024-07-27T00:28:34.389049213Z File "", line 1140, in _find_and_load_unlocked 2024-07-27T00:28:34.389054113Z ModuleNotFoundError: No module named 'main' 2024-07-27T00:28:34.398361536Z [2024-07-27 00:28:34 +0000] [68] [INFO] Worker exiting (pid: 68) 2024-07-27T00:28:34.658491279Z [2024-07-27 00:28:34 +0000] [63] [ERROR] Worker (pid:68) exited with code 3 2024-07-27T00:28:34.669995331Z [2024-07-27 00:28:34 +0000] [63] [ERROR] Shutting down: Master 2024-07-27T00:28:34.671412050Z [2024-07-27 00:28:34 +0000] [63] [ERROR] Reason: Worker failed to boot.

cforce avatar Jul 27 '24 09:07 cforce

related

  • https://github.com/Azure-Samples/azure-search-openai-demo/issues/1624
  • https://github.com/Azure-Samples/azure-search-openai-demo/issues/1361
  • https://github.com/Azure-Samples/azure-search-openai-demo/issues/95

Which gave me the rteleavt hints Insights > Investigate > Failures, and then filtering by recency.

Traceback (most recent call last): File "/agents/python/sitecustomize.py", line 97, in _load_instrumentors distro.load_instrumentor(entry_point, skip_dep_check=True) File "/agents/python/opentelemetry/instrumentation/distro.py", line 62, in load_instrumentor instrumentor: BaseInstrumentor = entry_point.load() ^^^^^^^^^^^^^^^^^^ File "/agents/python/pkg_resources/init.py", line 2473, in load self.require(*args, **kwargs) File "/agents/python/pkg_resources/init.py", line 2497, in require items = working_set.resolve(reqs, env, installer, extras=self.extras) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/agents/python/pkg_resources/init.py", line 787, in resolve dist = self._resolve_dist( ^^^^^^^^^^^^^^^^^^^ File "/agents/python/pkg_resources/init.py", line 833, in _resolve_dist raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.VersionConflict: (opentelemetry-util-http 0.45b0 (/agents/python), Requirement.parse('opentelemetry-util-http==0.46b0'))

Tried up upgrade to 0.47b0 but..

ERROR: Cannot install opentelemetry-instrumentation-aiohttp-client==0.46b0 and opentelemetry-util-http==0.45b0 because these package versions have conflicting dependencies. ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

-- for 0.45b0 i get

ERROR: Cannot install azure-monitor-opentelemetry==1.6.0 and opentelemetry-instrumentation-django==0.45b0 because these package versions have conflicting dependencies. ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

cforce avatar Jul 27 '24 12:07 cforce

There is some issue with dep - maybe

opentelemetry-semantic-conventions==0.46b0 opentelemetry-semantic-conventions-ai==0.3.5

cforce avatar Jul 29 '24 15:07 cforce

requirements.txt

ERROR: Cannot install -r app/backend/requirements.txt (line 242), -r app/backend/requirements.txt (line 244), -r app/backend/requirements.txt (line 248), -r app/backend/requirements.txt (line 250), -r app/backend/requirements.txt (line 252), -r app/backend/requirements.txt (line 256), -r app/backend/requirements.txt (line 258) and opentelemetry-instrumentation==0.47b0 because these package versions have conflicting dependencies. ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts Failed to restore backend python packages

cforce avatar Jul 30 '24 09:07 cforce

https://github.com/traceloop/openllmetry/issues/1694 https://github.com/open-telemetry/opentelemetry-python-contrib/issues/2752

cforce avatar Jul 30 '24 09:07 cforce

@cforce , did you find any solution for this issue. I am also facing the same.

abhiaryz avatar Sep 02 '24 07:09 abhiaryz

I think it was a problem with python deps / version of opentelemetry libs . Shall be fixed meanwhile

Am Mo., 2. Sept. 2024 um 09:19 Uhr schrieb Abhishek Kumar < @.***>:

@cforce https://github.com/cforce , did you find any solution for this issue. I am also facing the same.

— Reply to this email directly, view it on GitHub https://github.com/Azure-Samples/azure-search-openai-demo/issues/1864#issuecomment-2323992777, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADAARRJ3DQLKMWVHBVJBCTZUQGRJAVCNFSM6AAAAABLRXIYVKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRTHE4TENZXG4 . You are receiving this because you were mentioned.Message ID: @.***>

cforce avatar Sep 02 '24 10:09 cforce

how to fix this? do I azd up?

hwvm avatar Sep 05 '24 07:09 hwvm

For my team this is quite a problematic bug as we’re looking to take the app into production.

jeannot-prvlimburg avatar Oct 24 '24 14:10 jeannot-prvlimburg

For my team this is quite a problematic bug as we’re looking to take the app into production.

I have two implementations of this repo with different adjustments in the code, and this error occurred randomly in both of them at the same time.

jeannot-prvlimburg avatar Oct 24 '24 14:10 jeannot-prvlimburg

check https://github.com/Azure-Samples/azure-search-openai-demo/issues/2099

cforce avatar Oct 30 '24 23:10 cforce

The whole open telemetry domain is very sensitive and fast failing. Again i run in a similar problem in this corner i have the feeling that all otel and integration deps nee strict pinning of version which are tested together with the deps which are instrumented

https://github.com/Azure-Samples/azure-search-openai-demo/issues/2487

cforce avatar Apr 13 '25 08:04 cforce