authentik icon indicating copy to clipboard operation
authentik copied to clipboard
verified publisher icon goauthentik

Authentik no longer starts (critical event=WORKER TIMEOUT)

Open dimmenhau opened this issue 1 year ago • 3 comments

Describe the bug After the update from 2024.2.1 to 2024.2.2 Authentik no longer started

The message is thrown in the log file:

critical event=WORKER TIMEOUT (pid:158) logger=gunicorn.error timestamp=1710656355.5416307
critical event=WORKER TIMEOUT (pid:159) logger=gunicorn.error timestamp=1710656355.6090446
ERR event=WORKER (pid:158) was sent SIGKILL! Maybe the memory is full? logger=gunicorn.error timestamp=1710656356.861213
ERR event=Worker (pid:159) was sent SIGKILL! Possibly no more memory available? logger=gunicorn.error timestamp=1710656356.9548182

Logs it's a loop

INF event=Booting worker with pid: 158 logger=gunicorn.error timestamp=1710656325.3262706
INF event=Booting worker with pid: 159 logger=gunicorn.error timestamp=1710656325.4055312
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
INF domain_url=null event=Loaded MMDB database file=/geoip/GeoLite2-ASN.mmdb last_write=1710580833.924397 logger=authentik.events.context_processors.mmdb pid=158 schema_name=public timestamp=2024-03-17T06:18:57.831049
INF domain_url=null event=Loaded MMDB database file=/geoip/GeoLite2-City.mmdb last_write=1710580832.6405768 logger=authentik.events.context_processors.mmdb pid=158 schema_name=public timestamp=2024-03-17T06:18:57.841685
INF domain_url=null event=Loaded MMDB database file=/geoip/GeoLite2-ASN.mmdb last_write=1710580833.924397 logger=authentik.events.context_processors.mmdb pid=159 schema_name=public timestamp=2024-03-17T06:18:57.932980
INF domain_url=null event=Loaded MMDB database file=/geoip/GeoLite2-City.mmdb last_write=1710580832.6405768 logger=authentik.events.context_processors.mmdb pid=159 schema_name=public timestamp=2024-03-17T06:18:57.944044
INF app_name=authentik.tenants domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.tenants.checks pid=158 schema_name=public timestamp=2024-03-17T06:19:02.058794
/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py:98: RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
  warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
INF app_name=authentik.tenants domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.tenants.checks pid=159 schema_name=public timestamp=2024-03-17T06:19:02.307266
/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py:98: RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
  warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
INF app_name=authentik.admin domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.admin.tasks pid=158 schema_name=public timestamp=2024-03-17T06:19:02.883229
INF app_name=authentik.admin domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.admin.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:02.888976
INF app_name=authentik.crypto domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.crypto.tasks pid=158 schema_name=public timestamp=2024-03-17T06:19:02.912000
INF app_name=authentik.admin domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.admin.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:03.119449
INF app_name=authentik.admin domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.admin.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:03.124236
INF app_name=authentik.crypto domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.crypto.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:03.147143
INF app_name=authentik.flows domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.flows.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:06.651353
INF app_name=authentik.outposts domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.outposts.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:07.323989
INF app_name=authentik.outposts domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.outposts.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:07.333505
INF app_name=authentik.flows domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.flows.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:07.414160
INF domain_url=null event=Task published logger=authentik.root.celery pid=159 schema_name=public task_id=827b2c962d2746f2bfbde1df3b5f26da task_name=authentik.outposts.tasks.outpost_post_save timestamp=2024-03-17T06:19:08.075525
INF app_name=authentik.policies.reputation domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.reputation.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:08.107209
INF app_name=authentik.outposts domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.outposts.tasks pid=158 schema_name=public timestamp=2024-03-17T06:19:08.119491
INF app_name=authentik.policies.reputation domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.reputation.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:08.132097
INF app_name=authentik.outposts domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.outposts.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:08.160540
INF app_name=authentik.policies domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:08.444510
INF app_name=authentik.providers.proxy domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.proxy.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:08.495494
INF app_name=authentik.providers.proxy domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.proxy.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:08.520776
INF domain_url=null event=Task published logger=authentik.root.celery pid=158 schema_name=public task_id=930158016e2d430d94bb9b2be0eb2d80 task_name=authentik.outposts.tasks.outpost_post_save timestamp=2024-03-17T06:19:09.769549
INF app_name=authentik.policies.reputation domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.reputation.tasks pid=158 schema_name=public timestamp=2024-03-17T06:19:09.793217
INF app_name=authentik.policies.reputation domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.reputation.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:09.806646
INF app_name=authentik.policies domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:10.037403
INF app_name=authentik.providers.proxy domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.proxy.tasks pid=158 schema_name=public timestamp=2024-03-17T06:19:10.071385
INF app_name=authentik.providers.proxy domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.proxy.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:10.079154
warning error=authentik starting event=failed to proxy to backend logger=authentik.router timestamp=2024-03-17T06:19:12Z
INF app_name=authentik.providers.scim domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.scim.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:13.482069
INF app_name=authentik.providers.scim domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.scim.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:13.498979
INF app_name=authentik.rbac domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.rbac.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:13.537696
INF app_name=authentik.sources.ldap domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.sources.ldap.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:13.707631
INF app_name=authentik.sources.ldap domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.sources.ldap.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:13.747541
INF app_name=authentik.sources.oauth domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.sources.oauth.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:13.785977
/ak-root/venv/lib/python3.12/site-packages/facebook/__init__.py:99: SyntaxWarning: invalid escape sequence '\d'
 version_regex = re.compile("^\d\.\d{1,2}$")
INF app_name=authentik.sources.saml domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.sources.saml.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:14.078541
INF app_name=authentik.stages.authenticator_duo domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.stages.authenticator_duo.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:14.125867
INF app_name=authentik.stages.authenticator_static domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.stages.authenticator_static.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:14.170462
INF app_name=authentik.stages.email domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.stages.email.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:14.208297
critical event=WORKER TIMEOUT (pid:158) logger=gunicorn.error timestamp=1710656355.5416307
critical event=WORKER TIMEOUT (pid:159) logger=gunicorn.error timestamp=1710656355.6090446
ERR event=Worker (pid:158) was sent SIGKILL! Perhaps out of memory? logger=gunicorn.error timestamp=1710656356.861213
ERR event=Worker (pid:159) was sent SIGKILL! Perhaps out of memory? logger=gunicorn.error timestamp=1710656356.9548182
INF event=Booting worker with pid: 173 logger=gunicorn.error timestamp=1710656357.0165977
INF event=Booting worker with pid: 174 logger=gunicorn.error timestamp=1710656357.1532755
**Version and Deployment (please complete the following information):**
  • authentik version: 2024.2.2
  • Deployment: [e.g. docker-compose, helm]
version: "3.4"
services:
  postgresql:
    container_name: postgresql
    image: docker.io/library/postgres:12-alpine
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 5s
    volumes:
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/database:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: ${PG_PASS:?database password required}
      POSTGRES_USER: ${PG_USER:-authentik}
      POSTGRES_DB: ${PG_DB:-authentik}
    env_file:
      - .env
  redis:
    container_name: redis
    image: docker.io/library/redis:alpine
    command: --save 60 1 --loglevel warning
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 3s
    volumes:
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/redis:/data
  server:
    container_name: authentik
    image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2024.2.2}
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_REDIS__HOST: redis
      AUTHENTIK_POSTGRESQL__HOST: postgresql
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
    volumes:
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/media:/media
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/custom-templates:/templates
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/geoip:/geoip
    env_file:
      - .env
    ports:
      - "${COMPOSE_PORT_HTTP:-9000}:9000"
      - "${COMPOSE_PORT_HTTPS:-9443}:9443"
    depends_on:
      - postgresql
      - redis
      - geoip
  worker:
    container_name: authentik-worker
    image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2024.2.2}
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_REDIS__HOST: redis
      AUTHENTIK_POSTGRESQL__HOST: postgresql
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
    # `user: root` and the docker socket volume are optional.
    # See more for the docker socket integration here:
    # https://goauthentik.io/docs/outposts/integrations/docker
    # Removing `user: root` also prevents the worker from fixing the permissions
    # on the mounted folders, so when removing this make sure the folders have the correct UID/GID
    # (1000:1000 by default)
    # user: root
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/media:/media
      - ${DOCKER_VOLUME_CERT_STORAGE:-/mnt/docker-volumes}/certs:/certs
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/custom-templates:/templates
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/geoip:/geoip
    env_file:
      - .env
    depends_on:
      - postgresql
      - redis
      - geoip
  geoip:
    container_name: geoip
    image: "maxmindinc/geoipupdate:latest"
    restart: unless-stopped
    env_file:
      - .env-geoip
    volumes:
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/geoip:/usr/share/GeoIP

Additional context However, if I set the parameter AUTHENTIK_DEBUG=true, the error does not occur and Authentik starts

dimmenhau avatar Mar 17 '24 06:03 dimmenhau

I got the exact same issue, AUTHENTIK_DEBUG=true made this works! Using last version of the docker-compose.yml on the website, only change are the ports (https 9443 was already in use).

pathetiq avatar Apr 02 '24 17:04 pathetiq

The // in the regex pattern aren't escaped properly in the opencontainers dependency https://github.com/goauthentik/authentik/blob/main/poetry.lock#L2594. it's causing me the same issue too, but just started looking into it.

brentonmallen1 avatar May 09 '24 01:05 brentonmallen1

nvm, it seems to just be a warning but I went and made a pr to hopefully address it anyway 🤷

brentonmallen1 avatar May 09 '24 03:05 brentonmallen1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

authentik-automation[bot] avatar Jul 09 '24 01:07 authentik-automation[bot]