api-engine-tasks pod in CrashLoopBackOff: No module named 'api_engine'

[gabrieleCacciola]

I am trying to spin up your latest version of Cello on a kubernetes EKS cluster. I followed all the steps but I get your api-engine-tasks pod in CrashLoopBackOff with the following error:

Expected Behavior

All pods successfully deployed and running.

Current Behavior

default api-engine-99c857b95-6sgc7 1/1 Running 0 16s default api-engine-tasks-785f8cc46f-cwjtt 0/1 CrashLoopBackOff 1 17s default cello-nginx-85c45dcd6c-n6wpt 1/1 Running 0 14s default postgres-58d84fcb94-rwx8g 1/1 Running 0 19s default redis-6fbb95f5c8-9csf8 1/1 Running 0 18s

$ kubectl logs -f api-engine-tasks-785f8cc46f-cwjtt /scripts/change_local_settings.sh: line 5: /var/www/server/api_engine/settings.py.example: No such file or directory All initial Done!! Traceback (most recent call last): File "/usr/local/bin/celery", line 8, in sys.exit(main()) File "/usr/local/lib/python3.6/site-packages/celery/main.py", line 16, in main _main() File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 322, in main cmd.execute_from_commandline(argv) File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline super(CeleryCommand, self).execute_from_commandline(argv))) File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 273, in execute_from_commandline argv = self.setup_app_from_commandline(argv) File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 479, in setup_app_from_commandline self.app = self.find_app(app) File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 501, in find_app return find_app(app, symbol_by_name=self.symbol_by_name) File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 359, in find_app sym = symbol_by_name(app, imp=imp) File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 504, in symbol_by_name return imports.symbol_by_name(name, imp=imp) File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 57, in symbol_by_name module = imp(module_name, package=package, **kwargs) File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 104, in import_from_cwd return imp(module, package=package) File "/usr/local/lib/python3.6/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 953, in _find_and_load_unlocked ModuleNotFoundError: No module named 'api_engine'

Possible Solution

Steps to Reproduce

1. export DEPLOY_METHOD=k8s
2. export KUBECONFIG=/path/to/kubeconfig
3. export AWS_PROFILE=yourAWSprofile
4. make start

Context (Environment)

• OS: Mac
• Docker version: Client: Docker Engine - Community Version: 19.03.4 API version: 1.40 Go version: go1.12.10 Git commit: 9013bf5 Built: Thu Oct 17 23:44:48 2019 OS/Arch: darwin/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.4 API version: 1.40 (minimum version 1.12) Go version: go1.12.10 Git commit: 9013bf5 Built: Thu Oct 17 23:50:38 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.2.10 GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339 runc: Version: 1.0.0-rc8+dev GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 docker-init: Version: 0.18.0 GitCommit: fec3683

• Docker compose version: docker-compose version 1.24.1, build 4667896b docker-py version: 3.7.3 CPython version: 3.6.8 OpenSSL version: OpenSSL 1.1.0j 20 Nov 2018
• Kubernetes version: Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.7", GitCommit:"8fca2ec50a6133511b771a11559e24191b1aa2b4", GitTreeState:"clean", BuildDate:"2019-09-18T14:47:22Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-eks-5047ed", GitCommit:"5047edce664593832e9b889e447ac75ab104f527", GitTreeState:"clean", BuildDate:"2019-08-21T22:32:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
• Python version: $ python --version Python 2.7.10 $ python3 --version Python 3.7.3

[IvanKhramov]

Hi, I have exactly the same problem when I try to install a Cello into a cluster.

[marcosrachid]

I'm with the same problem when installing on the default namespace, but when I try to set a namespace by running:

K8S_DEPLOY_NAMESPACE=hyperledger-tools DEPLOY_METHOD=k8s make start

I'm receiving another error:

kubectl get pods -n hyperledger-tools

NAME READY STATUS RESTARTS AGE api-engine-99c857b95-wdkw9 1/1 Running 0 17s api-engine-tasks-5ff5ff755-pjp5h 0/1 CrashLoopBackOff 1 19s cello-nginx-85c45dcd6c-dvvc5 1/1 Running 0 15s postgres-58d84fcb94-ht6c8 1/1 Running 0 23s redis-6fbb95f5c8-hn8pn 1/1 Running 0 21s

kubectl describe pod api-engine-tasks-5ff5ff755-pjp5h -n hyperledger-tools

... Events: Type Reason Age From Message

---

Normal Scheduled 66s default-scheduler Successfully assigned hyperledger-tools/api-engine-tasks-5ff5ff755-pjp5h to aks-nodepool1-17372329-vmss000000 Warning Failed 65s kubelet, aks-nodepool1-17372329-vmss000000 Error: configmap "api-engine-config" not found Normal Pulled 18s (x5 over 65s) kubelet, aks-nodepool1-17372329-vmss000000 Container image "hyperledger/cello-api-engine" already present on machine Normal Created 17s (x4 over 64s) kubelet, aks-nodepool1-17372329-vmss000000 Created container api-engine-tasks Normal Started 17s (x4 over 63s) kubelet, aks-nodepool1-17372329-vmss000000 Started container api-engine-tasks Warning BackOff 2s (x6 over 60s) kubelet, aks-nodepool1-17372329-vmss000000 Back-off restarting failed container

kubectl get cm -n hyperledger-tools

NAME DATA AGE api-engine-config 13 4m3s nginx-config 2 4m1s postgres-config 3 4m9s

Any solution for these problems ?

[iamksseo]

I have exactly the same problem

# kubectl logs -f api-engine-76bf9df745-q7hxh
Generating the settings.py for api_engine
Traceback (most recent call last):
  File "manage.py", line 18, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 345, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 76, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 63, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 142, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 674, in exec_module
  File "<frozen importlib._bootstrap_external>", line 781, in get_code
  File "<frozen importlib._bootstrap_external>", line 741, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/www/server/api_engine/settings.py", line 27
    DEBUG =
           ^
SyntaxError: invalid syntax
Traceback (most recent call last):
  File "manage.py", line 18, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 345, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 76, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 63, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 142, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 674, in exec_module
  File "<frozen importlib._bootstrap_external>", line 781, in get_code
  File "<frozen importlib._bootstrap_external>", line 741, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/www/server/api_engine/settings.py", line 27
    DEBUG =
           ^
SyntaxError: invalid syntax
[uWSGI] getting INI configuration from /etc/uwsgi/apps-enabled/server.ini
open("./python_plugin.so"): No such file or directory [core/utils.c line 3724]
!!! UNABLE to load uWSGI plugin: ./python_plugin.so: cannot open shared object file: No such file or directory !!!