Azure
00-anomaly-detection-tutorial: Container fails to start on ACI
Tutorial: 00-anomaly-detection-tutorial Step: 5: Test model on Azure Container Instance
Issue: Container failed to start on ACI. Error log:
2020-06-28T11:55:11,852765769+00:00 - rsyslog/run
2020-06-28T11:55:11,852562969+00:00 - iot-server/run
2020-06-28T11:55:11,864671678+00:00 - nginx/run
2020-06-28T11:55:11,865188479+00:00 - gunicorn/run
EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting...
2020-06-28T11:55:12,130239579+00:00 - iot-server/finish 1 0
2020-06-28T11:55:12,135812284+00:00 - Exit code 1 is normal. Not restarting iot-server.
Starting gunicorn 19.9.0
Listening at: http://127.0.0.1:31311 (11)
Using worker: sync
worker timeout is set to 300
Booting worker with pid: 43
Exception in worker process
Traceback (most recent call last):
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "/var/azureml-server/wsgi.py", line 1, in
I am also facing this issue. https://stackoverflow.com/a/61895560/1180117
Hi so using print(ws.webservices['mysvc'].get_logs())
I get the same error
2020-09-01T13:06:56,682182228+00:00 - iot-server/run 2020-09-01T13:06:56,683466724+00:00 - rsyslog/run 2020-09-01T13:06:56,713681736+00:00 - gunicorn/run 2020-09-01T13:06:56,721711113+00:00 - nginx/run
EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting...
2020-09-01T13:06:56,996583013+00:00 - iot-server/finish 1 0 2020-09-01T13:06:57,004199091+00:00 - Exit code 1 is normal. Not restarting iot-server. Starting gunicorn 19.9.0
Listening at: http://127.0.0.1:31311 (13)
Using worker: sync
worker timeout is set to 300
Booting worker with pid: 45
Exception in worker process
Traceback (most recent call last): File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker worker.init_process() File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process self.load_wsgi() File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi self.wsgi = self.app.wsgi() File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load return self.load_wsgiapp() File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp return util.import_app(self.app_uri) File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app import(module) File "/var/azureml-server/wsgi.py", line 1, in
I am also getting the same error ModuleNotFoundError: No module named 'ruamel' after deploying the ACI webservice. From a quick internet search, I found that this is a known issue (https://docs.microsoft.com/en-us/azure/machine-learning/resource-known-issues#installation-and-import):
However, it is not clear how we can apply the suggested fix to the anomaly-detection-tutorial notebook.
@kawo123 yes same issue
Output log shows -
import ruamel.yaml ModuleNotFoundError: No module named 'ruamel' Worker exiting (pid: 45) Shutting down: Master Reason: Worker failed to boot. 2020-09-10T06:55:03,932157139+00:00 - gunicorn/finish 3 0 2020-09-10T06:55:03,933713250+00:00 - Exit code 3 is not normal. Killing image.
So pip install --upgrade azureml-sdk[notebooks,automl] --ignore-installed PyYAML needs to be added as requirement to the docker build step.
Create docker Image
Create docker Image This specifies the dependencies to include in the environment
from azureml.core.conda_dependencies import CondaDependencies
myenv = CondaDependencies.create(conda_packages=['pandas', 'scikit-learn', 'numpy'])
with open("myenv.yml","w") as f: f.write(myenv.serialize_to_string())
Included ruamel & ruamel.yaml as CondaDependencies while creating the docker image like below:
This specifies the dependencies to include in the environment
from azureml.core.conda_dependencies import CondaDependencies
myenv = CondaDependencies.create(conda_packages=['pandas', 'scikit-learn', 'numpy','ruamel','ruamel.yaml'])
with open("myenv.yml","w") as f: f.write(myenv.serialize_to_string())
This fixed the issue for me related to deploying the ACI web service with the created image.
