papermerge icon indicating copy to clipboard operation
papermerge copied to clipboard
verified publisher icon ciur

Backend API not responding to http request with kubernetes

Open chrispnugget88 opened this issue 3 years ago • 1 comments

Discription

Docker Container Image: papermerge/papermerge:2.1.2

I am having an issue deploying the BACKEND service in kubernetes. The pod starts and the server goes through all the motions with not much issues but when the /api/ request is sent only a 404 error is returned. The backend service uses all default configurations to my knowlege with the exception of modified ENV file.

All other services post sucsessfully:

  • FRONTEND [papermerge/papermerge.js:2.1.1]
  • BACKEND [papermerge/papermerge:2.1.2] [init /run.bash init] [cmd /run.bash server]
  • WORKER [papermerge/papermerge:2.1.2] [init /run.bash migrate] [cmd /run.bash worker]
  • WSSERVER [papermerge/papermerge:2.1.2] [cmd /run.bash ws_server]

Expected

Http response without error when requesting /api/.

spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 12, cores: 4)
spawned uWSGI worker 2 (pid: 16, cores: 4)
[pid: 16|app: 0|req: 1/1] 10.52.0.1 () {26 vars in 316 bytes} [Thu Dec 29 17:03:38 2022] GET /api/ => generated 539 bytes in 302 msecs (HTTP/1.1 200) 7 headers in 223 bytes (1 switches on core 0)
[pid: 16|app: 0|req: 2/2] 10.52.0.1 () {26 vars in 316 bytes} [Thu Dec 29 17:03:38 2022] GET /api/ => generated 539 bytes in 3 msecs (HTTP/1.1 200) 7 headers in 223 bytes (1 switches on core 1)
[pid: 12|app: 0|req: 1/3] 10.52.0.1 () {26 vars in 316 bytes} [Thu Dec 29 17:03:38 2022] GET /api/ => generated 539 bytes in 310 msecs (HTTP/1.1 200) 7 headers in 223 bytes (1 switches on core 0)
[pid: 16|app: 0|req: 3/4] 10.52.0.1 () {26 vars in 316 bytes} [Thu Dec 29 17:03:48 2022] GET /api/ => generated 539 bytes in 3 msecs (HTTP/1.1 200) 7 headers in 223 bytes (1 switches on core 2)

Actual

Http returns 404 error when requesting /api/

spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 34, cores: 4)
spawned uWSGI worker 2 (pid: 38, cores: 4)
WARNING 2022-12-29 17:08:16,328 log Not Found: /api/
WARNING 2022-12-29 17:08:16,328 log Not Found: /api/
[pid: 34|app: 0|req: 2/1] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 17:08:16 2022] GET /api/ => generated 179 bytes in 250 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 0)
[pid: 34|app: 0|req: 2/2] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 17:08:16 2022] GET /api/ => generated 179 bytes in 250 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 2)
WARNING 2022-12-29 17:08:16,701 log Not Found: /api/
[pid: 38|app: 0|req: 1/3] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 17:08:16 2022] GET /api/ => generated 179 bytes in 371 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 0)
Backend Service Full Log (CLICK TO VIEW) 
INFO 2022-12-29 16:24:15,347 <frozen importlib pikepdf C++ to Python logger bridge initialized
/venv/lib/python3.10/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (None)/charset_normalizer (3.0.1) doesn't match a supported version!
  warnings.warn(

93 static files copied to '/app/static'.
INFO 2022-12-29 16:24:17,088 <frozen importlib pikepdf C++ to Python logger bridge initialized
/venv/lib/python3.10/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (None)/charset_normalizer (3.0.1) doesn't match a supported version!
  warnings.warn(
Operations to perform:
  Apply all migrations: auth, authtoken, contenttypes, core, dynamic_preferences, dynamic_preferences_users, knox, sessions, sites
Running migrations:
  No migrations to apply.
Creating superuser username=admin
INFO 2022-12-29 16:24:19,276 <frozen importlib pikepdf C++ to Python logger bridge initialized
/venv/lib/python3.10/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (None)/charset_normalizer (3.0.1) doesn't match a supported version!
  warnings.warn(
CommandError: Error: That username is already taken.
INFO 2022-12-29 16:24:21,343 <frozen importlib pikepdf C++ to Python logger bridge initialized
/venv/lib/python3.10/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (None)/charset_normalizer (3.0.1) doesn't match a supported version!
  warnings.warn(
Indexing 0 Documents
Indexing 2 Folders
[uWSGI] getting INI configuration from /etc/uwsgi/papermerge.ini
[uwsgi-static] added mapping for /static => /app/static
*** Starting uWSGI 2.0.21 (64bit) on [Thu Dec 29 16:24:22 2022] ***
compiled with version: 10.2.1 20210110 on 24 December 2022 07:22:51
os: Linux-5.19.0-26-generic #27-Ubuntu SMP PREEMPT_DYNAMIC Wed Nov 23 20:44:15 UTC 2022
nodename: papermerge-backend-8568cd98b9-czvm2
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /app
detected binary path: /venv/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
building mime-types dictionary from file /etc/mime.types...1476 entry found
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address :8000 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.10.9 (main, Dec 21 2022, 08:51:48) [GCC 10.2.1 20210110]
PEP 405 virtualenv detected: /venv
Set PythonHome to /venv
Python main interpreter initialized at 0x559fcf484170
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 312864 bytes (305 KB) for 8 cores
*** Operational MODE: preforking+threaded ***
INFO 2022-12-29 16:24:23,508 <frozen importlib pikepdf C++ to Python logger bridge initialized
/venv/lib/python3.10/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (None)/charset_normalizer (3.0.1) doesn't match a supported version!
  warnings.warn(
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x559fcf484170 pid: 1 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 34, cores: 4)
spawned uWSGI worker 2 (pid: 38, cores: 4)
WARNING 2022-12-29 16:24:24,148 log Not Found: /api/
[pid: 34|app: 0|req: 3/1] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:24:23 2022] GET /api/ => generated 179 bytes in 289 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 2)
WARNING 2022-12-29 16:24:24,150 log Not Found: /api/
[pid: 34|app: 0|req: 3/2] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:24:23 2022] GET /api/ => generated 179 bytes in 290 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 1)
WARNING 2022-12-29 16:24:24,151 log Not Found: /api/
[pid: 34|app: 0|req: 3/3] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:24:23 2022] GET /api/ => generated 179 bytes in 292 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 0)
WARNING 2022-12-29 16:24:32,462 log Not Found: /api/
[pid: 34|app: 0|req: 4/4] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:24:32 2022] GET /api/ => generated 179 bytes in 2 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 3)
WARNING 2022-12-29 16:24:32,713 log Not Found: /api/
[pid: 38|app: 0|req: 1/5] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:24:32 2022] GET /api/ => generated 179 bytes in 253 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 0)
WARNING 2022-12-29 16:24:42,462 log Not Found: /api/
[pid: 34|app: 0|req: 5/6] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:24:42 2022] GET /api/ => generated 179 bytes in 2 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 2)
WARNING 2022-12-29 16:24:42,463 log Not Found: /api/
[pid: 38|app: 0|req: 2/7] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:24:42 2022] GET /api/ => generated 179 bytes in 2 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 1)
WARNING 2022-12-29 16:24:52,462 log Not Found: /api/
WARNING 2022-12-29 16:24:52,462 log Not Found: /api/
[pid: 34|app: 0|req: 6/8] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:24:52 2022] GET /api/ => generated 179 bytes in 2 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 1)
[pid: 38|app: 0|req: 3/9] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:24:52 2022] GET /api/ => generated 179 bytes in 2 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 2)
WARNING 2022-12-29 16:25:02,463 log Not Found: /api/
WARNING 2022-12-29 16:25:02,463 log Not Found: /api/
[pid: 34|app: 0|req: 7/10] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:25:02 2022] GET /api/ => generated 179 bytes in 2 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 0)
[pid: 38|app: 0|req: 4/11] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:25:02 2022] GET /api/ => generated 179 bytes in 2 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 3)
WARNING 2022-12-29 16:25:12,462 log Not Found: /api/
WARNING 2022-12-29 16:25:12,462 log Not Found: /api/
[pid: 38|app: 0|req: 5/13] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:25:12 2022] GET /api/ => generated 179 bytes in 1 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 0)
[pid: 34|app: 0|req: 8/13] 10.52.0.1 () {26 vars in 315 bytes} [Thu Dec 29 16:25:12 2022] GET /api/ => generated 179 bytes in 1 msecs (HTTP/1.1 404) 7 headers in 213 bytes (1 switches on core 3)

Info

SYSTEM 
-----INFO-----
OS=UBUNTU 22.10
KUBERNETES_VERSION=v1.24.9-rc3
TRAEFIK_VERSION=2.9.4
DOCKER_SERVER_VERSION=20.10.21
DOCKER_CLIENT_VERSION=20.10.21
POSTGRESQL_VERSION=15.1.0-debian-11-r12
KUBE DETAIL ENV 
-----KUBERNETES ENV-----
BACKEND_NODEPORT_PORT_8000_TCP_ADDR=10.53.193.93
BACKEND_NODEPORT_PORT_8000_TCP_PORT=8000
BACKEND_NODEPORT_PORT_8000_TCP_PROTO=tcp
BACKEND_NODEPORT_PORT=tcp://10.53.193.93:8000
BACKEND_NODEPORT_SERVICE_PORT_HTTP=8000
BACKEND_NODEPORT_SERVICE_PORT=8000
HOME=/root
HOSTNAME=papermerge-backend-8568cd98b9-czvm2
KUBERNETES_PORT_443_TCP_ADDR=10.53.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP=tcp://10.53.0.1:443
KUBERNETES_PORT=tcp://10.53.0.1:443
KUBERNETES_SERVICE_HOST=10.53.0.1
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
LANG=C.UTF-8
PAPERMERGE_BACKEND_PORT_8000_TCP_ADDR=10.53.75.186
PAPERMERGE_BACKEND_PORT_8000_TCP_PORT=8000
PAPERMERGE_BACKEND_PORT_8000_TCP_PROTO=tcp
PAPERMERGE_BACKEND_PORT_8000_TCP=tcp://10.53.75.186:8000
PAPERMERGE_BACKEND_PORT=tcp://10.53.75.186:8000
PAPERMERGE_BACKEND_SERVICE_HOST=10.53.75.186
PAPERMERGE_BACKEND_SERVICE_PORT_HTTP=8000
PAPERMERGE_BACKEND_SERVICE_PORT=8000
PAPERMERGE_FRONTEND_PORT_80_TCP_ADDR=10.53.87.236
PAPERMERGE_FRONTEND_PORT_80_TCP_PORT=80
PAPERMERGE_FRONTEND_PORT_80_TCP_PROTO=tcp
PAPERMERGE_FRONTEND_PORT_80_TCP=tcp://10.53.87.236:80
PAPERMERGE_FRONTEND_PORT=tcp://10.53.87.236:80
PAPERMERGE_FRONTEND_SERVICE_HOST=10.53.87.236
PAPERMERGE_FRONTEND_SERVICE_PORT_HTTP=80
PAPERMERGE_FRONTEND_SERVICE_PORT=80
PAPERMERGE_WORKER_PORT_80_TCP_ADDR=10.53.156.137
PAPERMERGE_WORKER_PORT_80_TCP_PORT=80
PAPERMERGE_WORKER_PORT_80_TCP_PROTO=tcp
PAPERMERGE_WORKER_PORT_80_TCP=tcp://10.53.156.137:80
PAPERMERGE_WORKER_PORT=tcp://10.53.156.137:80
PAPERMERGE_WORKER_SERVICE_HOST=10.53.156.137
PAPERMERGE_WORKER_SERVICE_PORT_HTTP=80
PAPERMERGE_WORKER_SERVICE_PORT=80
PAPERMERGE_WSSERVER_PORT_8000_TCP_ADDR=10.53.210.87
PAPERMERGE_WSSERVER_PORT_8000_TCP_PORT=8000
PAPERMERGE_WSSERVER_PORT_8000_TCP_PROTO=tcp
PAPERMERGE_WSSERVER_PORT_8000_TCP=tcp://10.53.210.87:8000
PAPERMERGE_WSSERVER_PORT=tcp://10.53.210.87:8000
PAPERMERGE_WSSERVER_SERVICE_HOST=10.53.210.87
PAPERMERGE_WSSERVER_SERVICE_PORT_HTTP=8000
PAPERMERGE_WSSERVER_SERVICE_PORT=8000
PATH=/venv/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/app
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/66030fa03382b4914d4c4d0896961a0bdeeeb274/public/get-pip.py
PYTHON_PIP_VERSION=22.3.1
PYTHON_SETUPTOOLS_VERSION=65.5.0
PYTHON_VERSION=3.10.9
PYTHONBUFFERED=1
REDIS_MASTER_PORT_6379_TCP_ADDR=10.53.231.246
REDIS_MASTER_PORT_6379_TCP_PORT=6379
REDIS_MASTER_PORT_6379_TCP_PROTO=tcp
REDIS_MASTER_PORT_6379_TCP=tcp://10.53.231.246:6379
REDIS_MASTER_PORT=tcp://10.53.231.246:6379
REDIS_MASTER_SERVICE_HOST=10.53.231.246
REDIS_MASTER_SERVICE_PORT_TCP_REDIS=6379
REDIS_MASTER_SERVICE_PORT=6379
TERM=xterm-256color
VIRTUAL_ENV=/venv
_=/usr/bin/env
PAPERMERGE ENV 
DJANGO_SETTINGS_MODULE=config.settings
[email protected]
DJANGO_SUPERUSER_PASSWORD=password
DJANGO_SUPERUSER_USERNAME=admin
PAPERMERGE__DATABASE__HOST=db.postgresql.svc.k3s.ganon.local
PAPERMERGE__DATABASE__NAME=papermerge
PAPERMERGE__DATABASE__PASSWORD=papermerge
PAPERMERGE__DATABASE__TYPE=postgres
PAPERMERGE__DATABASE__USER=papermerge
PAPERMERGE__MAIN__MEDIA_ROOT=/app/media
PAPERMERGE__MAIN__SECRET_KEY=SuperSecret
PAPERMERGE__MAIN__TIMEZONE=GMT
PAPERMERGE__REDIS__HOST=redis-master

chrispnugget88 avatar Dec 29 '22 17:12 chrispnugget88

Can you please describe your k8s setup so that I can reproduce the issue? What helm chart did you use? Did you deploy using minikube setup ? What values did you use in the helm chart ?

ciur avatar Jan 01 '23 12:01 ciur