502 after upgrading to latest

[bilyboy785]

Describe the bug After upgrading to latest on docker stack, app is unreachable

To Reproduce docker-compose pull && docker-compose up -d

Expected behavior App should be reachable

Logs

/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('
  "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"

/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py:98: RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
  warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)

/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('
  "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('
  "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('
  "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('
  "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"

Version and Deployment (please complete the following information):

• authentik version: 2024.2.0
• Deployment: docker-compose

[CanonCan]

I'm seeing same issue error 500 for all my containers using authentik

i reverted to beryju/authentik:2023.10.7 as the version and everything is working again

[bilyboy785]

I did the same, but i needed to restore my latest backup for authentik folders (postgres, authentik). For now all is good on 2023.10.7

[klutchrider]

Same, had to revert back to 2023.10.7

[sammcj]

Likewise, lots of errors in the logs, another broken release 😢

2024-02-22 23:56:17 [info     ] waiting to acquire database lock
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/lifecycle/migrate.py", line 112, in <module>
    execute_from_command_line(["", "migrate_schemas"])
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 382, in execute
    settings.INSTALLED_APPS
  File "/ak-root/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 89, in __getattr__
    self._setup(name)
  File "/ak-root/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 76, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 190, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/authentik/root/settings.py", line 431, in <module>
    "location": Path(CONFIG.get("storage.media.file.path")),
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/pathlib.py", line 1162, in __init__
    super().__init__(*args)
  File "/usr/local/lib/python3.12/pathlib.py", line 373, in __init__
    raise TypeError(
TypeError: argument should be a str or an os.PathLike object where __fspath__ returns a str, not 'NoneType'

[CanonCan]

I couldn't even load the logs it would just crash until I went back to the old one didn't have to restore anything just changed the version number and get started up

Sincerely,

Michael

On Feb 22, 2024 at 16:57, Sam @.***> wrote:

Likewise, lots of errors in the logs, looks like another untested release 😢

2024-02-22 23:56:17 [info ] waiting to acquire database lock Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/lifecycle/migrate.py", line 112, in execute_from_command_line(["", "migrate_schemas"]) File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/init.py", line 442, in execute_from_command_line utility.execute() File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/init.py", line 382, in execute settings.INSTALLED_APPS File "/ak-root/venv/lib/python3.12/site-packages/django/conf/init.py", line 89, in getattr self._setup(name) File "/ak-root/venv/lib/python3.12/site-packages/django/conf/init.py", line 76, in _setup self._wrapped = Settings(settings_module) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/ak-root/venv/lib/python3.12/site-packages/django/conf/init.py", line 190, in init mod = importlib.import_module(self.SETTINGS_MODULE) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1387, in _gcd_import File "", line 1360, in _find_and_load File "", line 1331, in _find_and_load_unlocked File "", line 935, in _load_unlocked File "", line 995, in exec_module File "", line 488, in _call_with_frames_removed File "/authentik/root/settings.py", line 431, in "location": Path(CONFIG.get("storage.media.file.path")), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/pathlib.py", line 1162, in init super().init(*args) File "/usr/local/lib/python3.12/pathlib.py", line 373, in init raise TypeError( TypeError: argument should be a str or an os.PathLike object where fspath returns a str, not 'NoneType'

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[sammcj]

Downgrade to 2023.10.7 doesn't work for me, crashes on db migrations now:

Operations to perform:
  Apply all migrations: auth, authentik_blueprints, authentik_core, authentik_crypto, authentik_enterprise, authentik_events, authentik_flows, authentik_outposts, authentik_policies, authentik_policies_dummy, authentik_policies_event_matcher, authentik_policies_expiry, authentik_policies_expression, authentik_policies_password, authentik_policies_reputation, authentik_providers_ldap, authentik_providers_oauth2, authentik_providers_proxy, authentik_providers_radius, authentik_providers_saml, authentik_providers_scim, authentik_rbac, authentik_sources_ldap, authentik_sources_oauth, authentik_sources_plex, authentik_sources_saml, authentik_stages_authenticator_duo, authentik_stages_authenticator_sms, authentik_stages_authenticator_static, authentik_stages_authenticator_totp, authentik_stages_authenticator_validate, authentik_stages_authenticator_webauthn, authentik_stages_captcha, authentik_stages_consent, authentik_stages_deny, authentik_stages_dummy, authentik_stages_email, authentik_stages_identification, authentik_stages_invitation, authentik_stages_password, authentik_stages_prompt, authentik_stages_user_delete, authentik_stages_user_login, authentik_stages_user_logout, authentik_stages_user_write, authentik_tenants, contenttypes, guardian, sessions
Running migrations:
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/lifecycle/migrate.py", line 112, in <module>
    execute_from_command_line(["", "migrate"])
  File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/migration.py", line 118, in apply
    operation.state_forwards(self.app_label, project_state)
  File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/operations/fields.py", line 93, in state_forwards
    state.add_field(
  File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/state.py", line 248, in add_field
    self.models[model_key].fields[name] = field
    ~~~~~~~~~~~^^^^^^^^^^^
KeyError: ('authentik_tenants', 'tenant')
  Applying authentik_tenants.0002_tenant_flow_user_settings...#

testing 2024.2.1 shows that it is also broken.

[CanonCan]

Weird didn't happen to me I just downgraded it started up and all my MFA protected sites were working again I was getting error 500 specifically on them before the downgrade

Sincerely,

Michael

On Feb 22, 2024 at 17:00, Sam @.***> wrote:

Downgrade to 2023.10.7 doesn't work for me, crashes on db migrations now:

Operations to perform: Apply all migrations: auth, authentik_blueprints, authentik_core, authentik_crypto, authentik_enterprise, authentik_events, authentik_flows, authentik_outposts, authentik_policies, authentik_policies_dummy, authentik_policies_event_matcher, authentik_policies_expiry, authentik_policies_expression, authentik_policies_password, authentik_policies_reputation, authentik_providers_ldap, authentik_providers_oauth2, authentik_providers_proxy, authentik_providers_radius, authentik_providers_saml, authentik_providers_scim, authentik_rbac, authentik_sources_ldap, authentik_sources_oauth, authentik_sources_plex, authentik_sources_saml, authentik_stages_authenticator_duo, authentik_stages_authenticator_sms, authentik_stages_authenticator_static, authentik_stages_authenticator_totp, authentik_stages_authenticator_validate, authentik_stages_authenticator_webauthn, authentik_stages_captcha, authentik_stages_consent, authentik_stages_deny, authentik_stages_dummy, authentik_stages_email, authentik_stages_identification, authentik_stages_invitation, authentik_stages_password, authentik_stages_prompt, authentik_stages_user_delete, authentik_stages_user_login, authentik_stages_user_logout, authentik_stages_user_write, authentik_tenants, contenttypes, guardian, sessions Running migrations: Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/lifecycle/migrate.py", line 112, in execute_from_command_line(["", "migrate"]) File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/init.py", line 442, in execute_from_command_line utility.execute() File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/init.py", line 436, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv self.execute(*args, **cmd_options) File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute output = self.handle(*args, **options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper res = handle_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle post_migrate_state = executor.migrate( ^^^^^^^^^^^^^^^^^ File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate state = self._migrate_all_forwards( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards state = self.apply_migration( ^^^^^^^^^^^^^^^^^^^^^ File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration state = migration.apply(state, schema_editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/migration.py", line 118, in apply operation.state_forwards(self.app_label, project_state) File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/operations/fields.py", line 93, in state_forwards state.add_field( File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/state.py", line 248, in add_field self.models[model_key].fields[name] = field ~~~~~~~~~~~^^^^^^^^^^^ KeyError: ('authentik_tenants', 'tenant') Applying authentik_tenants.0002_tenant_flow_user_settings...#

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[sammcj]

Fix it!

In my authentik config yml, I had to add:

storage:
  media:
    file:
      path: "/media"

The code that parses this must have a bug in it - https://github.com/goauthentik/authentik/blob/main/authentik/root/settings.py#L431

[rissson]

We indeed don't support downgrading because of database migrations. The proper way to upgrade is to backup first and restore the backup if something goes wrong.

As for the config changes, they shouldn't impact you unless you specifically overwrite /authentik/lib/default.yml in the container, which you shouldn't do. If you really want to use a configuration file instead of environment variables, you can mount it at /etc/authentik/config.yml (see https://github.com/goauthentik/authentik/blob/main/authentik/lib/config.py#L22 for details).

As for the initial issue, I'm afraid that the provided logs don't contain any information about the potential issue. Those lines are warnings coming from libraries we use that we don't have impact on, or that we haven't fixed yet as we were approaching this release.

[geokvant]

in my case for version 2024.2.1 chmod for folder media with user:users (according to my docker compose PUID and PGID) solved a problem after 3-5 minutes

[siankatabg]

The way I fixed it for myself was to add "user: root" to the "server" in my docker-compose. The idea came from this comment under the "worker" configs - "# Removing user: root also prevents the worker from fixing the permissions", so I've tried it on the "server" as the error I had was a permission issue with the "media" folder and it worked 👍

P.s. - Once everything started correctly, I've removed that line and it still works fine.

[patrick-GH]

The way I fixed it for myself was to add "user: root" to the "server" in my docker-compose. The idea came from this comment under the "worker" configs - "# Removing user: root also prevents the worker from fixing the permissions", so I've tried it on the "server" as the error I had was a permission issue with the "media" folder and it worked 👍

P.s. - Once everything started correctly, I've removed that line and it still works fine.

is --privileged=true the same as "user:root"

[siankatabg]

The way I fixed it for myself was to add "user: root" to the "server" in my docker-compose. The idea came from this comment under the "worker" configs - "# Removing user: root also prevents the worker from fixing the permissions", so I've tried it on the "server" as the error I had was a permission issue with the "media" folder and it worked 👍 P.s. - Once everything started correctly, I've removed that line and it still works fine.

is --privileged=true the same as "user:root"

No, I think those are different things, you can check the docker docs.

[mostdcoa]

I am getting the same error. No matter what I try (clean install, downgrade, upgrade, whatever) the result is the same. No DB (or any file for that matter) is created in any of the mapped directories.

One thing to note is the the wget https://goauthentik.io/docker-compose.yml pulls a docker-compose.yml file with 2023.10.7 tags in it vs 2024.2.2.

authentik-worker-1 | Operations to perform: authentik-worker-1 | Apply all migrations: auth, authentik_blueprints, authentik_core, authentik_crypto, authentik_enterprise, authentik_events, authentik_flows, authentik_outposts, authentik_policies, authentik_policies_dummy, authentik_policies_event_matcher, authentik_policies_expiry, authentik_policies_expression, authentik_policies_password, authentik_policies_reputation, authentik_providers_ldap, authentik_providers_oauth2, authentik_providers_proxy, authentik_providers_radius, authentik_providers_saml, authentik_providers_scim, authentik_rbac, authentik_sources_ldap, authentik_sources_oauth, authentik_sources_plex, authentik_sources_saml, authentik_stages_authenticator_duo, authentik_stages_authenticator_sms, authentik_stages_authenticator_static, authentik_stages_authenticator_totp, authentik_stages_authenticator_validate, authentik_stages_authenticator_webauthn, authentik_stages_captcha, authentik_stages_consent, authentik_stages_deny, authentik_stages_dummy, authentik_stages_email, authentik_stages_identification, authentik_stages_invitation, authentik_stages_password, authentik_stages_prompt, authentik_stages_user_delete, authentik_stages_user_login, authentik_stages_user_logout, authentik_stages_user_write, authentik_tenants, contenttypes, guardian, sessions authentik-worker-1 | Running migrations: authentik-worker-1 | Traceback (most recent call last): authentik-worker-1 | File "", line 198, in _run_module_as_main authentik-worker-1 | File "", line 88, in _run_code authentik-worker-1 | File "/lifecycle/migrate.py", line 112, in authentik-worker-1 | execute_from_command_line(["", "migrate"]) authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/init.py", line 442, in execute_from_command_line authentik-worker-1 | utility.execute() authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/init.py", line 436, in execute authentik-worker-1 | self.fetch_command(subcommand).run_from_argv(self.argv) authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv authentik-worker-1 | self.execute(*args, **cmd_options) authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute authentik-worker-1 | output = self.handle(*args, **options) authentik-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper authentik-worker-1 | res = handle_func(*args, **kwargs) authentik-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle authentik-worker-1 | post_migrate_state = executor.migrate( authentik-worker-1 | ^^^^^^^^^^^^^^^^^ authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate authentik-worker-1 | state = self._migrate_all_forwards( authentik-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards authentik-worker-1 | state = self.apply_migration( authentik-worker-1 | ^^^^^^^^^^^^^^^^^^^^^ authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration authentik-worker-1 | state = migration.apply(state, schema_editor) authentik-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/migration.py", line 118, in apply authentik-worker-1 | operation.state_forwards(self.app_label, project_state) authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/operations/fields.py", line 93, in state_forwards authentik-worker-1 | state.add_field( authentik-worker-1 | File "/ak-root/venv/lib/python3.11/site-packages/django/db/migrations/state.py", line 248, in add_field authentik-worker-1 | self.models[model_key].fields[name] = field authentik-worker-1 | ~~~~~~~~~~~^^^^^^^^^^^ authentik-worker-1 | KeyError: ('authentik_tenants', 'tenant') authentik-worker-1 | Sentry is attempting to send 2 pending events authentik-worker-1 | Waiting up to 2 seconds authentik-worker-1 | Press Ctrl-C to quit

[bilyboy785]

Hi, i'm getting this error again .... Logs are difficult to parse, so i can't find the root problem :(

Anyone can help on this ?

[mostdcoa]

@bilyboy785

I reported this issue, they said it wasn't an issue, you can try to set the volume path like I did and see if it helps.

https://github.com/goauthentik/authentik/issues/8864