zitadel [Bug]: --init-projections doesn't seem to work

Preflight Checklist

[X] I could not find a solution in the documentation, the existing issues or discussions
[ ] I have joined the ZITADEL chat

Environment

Self-hosted

Version

v2.44.0

Database

PostgreSQL

Database Version

16.1

Describe the problem caused by this bug

I hoped, the new --init-projections flag along with the new config block for the InitProjections would solve the bug migration already started, will check again in 5 seconds.

I tired it with and without using the flag, with and without the InitProjections block and with just setup as well as setup clearup. However, all of these commands result into the same bug like before - even setup clearup returns this now!!!:

root@myserver:/home# /usr/local/bin/zitadel setup --config /etc/zitadel.yaml --masterkey MY_KEY --tlsMode external
INFO[0000] setup started                                 caller="/home/runner/work/zitadel/zitadel/cmd/setup/setup.go:90"
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=14_events_push
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=01_tables
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=02_assets
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=03_default_instance
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=05_last_failed
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=06_resource_owner_columns
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=07_logstore
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=08_auth_token_indexes
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=12_auth_users_otp_columns
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=13_fix_quota_constraints
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=15_current_projection_state
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=16_unique_constraint_lower
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=17_add_offset_col_to_current_states
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=19_add_current_sequences_index
INFO[0001] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=20_add_by_user_index_on_session
INFO[0002] starting migration                            caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:66" name=20_add_by_user_index_on_session
INFO[0002] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=22_active_instance_events_index
INFO[0003] starting migration                            caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:66" name=22_active_instance_events_index
INFO[0005] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=config_change
INFO[0007] verify migration                              caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=projection_tables
WARN[0007] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0012] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0017] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0022] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0027] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0032] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0037] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0042] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0047] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0052] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0057] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0062] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0068] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0073] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0078] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0083] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables
WARN[0088] migration already started, will check again in 5 seconds  caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=projection_tables

To reproduce

Setup a old instance with v2.41.9 and PostgreSQL
Download the latest version v2.44.0
Run any setup command with or without --init-projections, with or without the block in config set to true or false, with or without the cleanup command
The result is the same in each case: migration already started, will check again in 5 seconds

Screenshots

No response

Expected behavior

I would expect no migrations to run when init projections are not enabled?!

Operating System

Ubuntu 22.04.3 LTS

Relevant Configuration

No response

Additional Context

No response

Jan 26 '24 10:01 CMiksche

Hi

We fixed zitadel setup cleanup so that it can cleanup all steps. You might have to run it multiple times until all steps are cleaned up because it manipulates only one step per run.

Jan 26 '24 12:01 adlerhurst

I just saw that you wrote cleaRup instead of cleaNup. So which of those did you use?

Jan 26 '24 13:01 adlerhurst

I just saw that you wrote cleaRup instead of cleaNup. So which of those did you use?

Ah, thank you. I used clearup instead of cleanup.

Now I can execute the migrations but afterwards it shows {"error":"invalid_request","error_description":"Errors.App.NotFound"}

The following is the error in the console:

time=2024-01-26T14:58:48.695+01:00 level=WARN msg="request error" oidc_error.parent="ID=QUERY-wu6Ee Message=Errors.App.NotFound Parent=(sql: no rows in result set)" oidc_error.description=Errors.App.NotFound oidc_error.type=invalid_request status_code=400
DEBU[0031] trigger iteration                             caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:417" iteration=0 projection=auth.refresh_tokens

Luckily the old v2.41.11 binary still works with the upgraded database

Jan 26 '24 14:01 CMiksche

After cleanup worked how did you setup or start-from-* zitadel? Including the flag or not?

Jan 26 '24 14:01 adlerhurst

After cleanup worked how did you setup or start-from-* zitadel? Including the flag or not?

I did the setup command like this:

zitadel setup --config /etc/zitadel.yaml --masterkey MYKEY --init-projections --tlsMode external

After that I started zitadel like this:

zitadel start --config /etc/zitadel.yaml --masterkey MYKEY --tlsMode external

Jan 26 '24 14:01 CMiksche

Hm ok that looks correct on the first view.

the setup completed successfully?

Jan 26 '24 17:01 adlerhurst

the setup completed successfully?

Yes - and if I run the command again it just verifies everything. So it looks like there are no Errors in that step, only afterwards when I actually want to start Zitadel.

Jan 26 '24 18:01 CMiksche

Can you please execute the following query, replace <CLIENT_ID> with your client id and send me the result so we can find out why the app isn't projected anymore?

with project as (
    select 
        a.project_id id,
        a.instance_id,
        a.resource_owner 
    from 
        projections.apps5_oidc_configs ao 
    join 
        projections.apps5 a 
        on a.id = ao.app_id 
        and a.instance_id = ao.instance_id 
    where 
        ao.client_id = '<CLIENT_ID>'
), project_events as (
    select 
        event_type , created_at
    from 
        eventstore.events2 
    where 
        aggregate_type = 'project' AND aggregate_id = (select id from project)
    order by position, in_tx_order
), org_events as (
    select 
        event_type , created_at
    from 
        eventstore.events2 
    where 
        aggregate_type = 'org' AND aggregate_id = (select resource_owner from project)
    order by position, in_tx_order
), instance_events as (
    select 
        event_type, created_at
    from 
        eventstore.events2 
    where 
        aggregate_type = 'instance' AND aggregate_id = (select instance_id from project)
    order by position, in_tx_order
)
Select * from project_events
UNION ALL
Select * from org_events
UNION ALL
Select * from instance_events;

Jan 29 '24 14:01 adlerhurst

Can you please execute the following query, replace <CLIENT_ID> with your client id and send me the result so we can find out why the app isn't projected anymore?

Of course. I used the Zitadel Console UI CLIENT_ID as the console shows the error message after the upgrade. Please note that the current state is the following: I ran the migrations but currently still use the old v2.41.11 binary because the new v2.44.0 binary shows me the error message.

I ran the same query with the Admin-, Auth- and Management-API Client Ids but the results were empty.

Now the output of the query:

event_type                                       |created_at                   |
-------------------------------------------------+-----------------------------+
project.added                                    |2022-12-08 13:48:32.111 +0100|
project.member.added                             |2022-12-08 13:48:32.111 +0100|
project.application.added                        |2022-12-08 13:48:32.111 +0100|
project.application.config.api.added             |2022-12-08 13:48:32.111 +0100|
project.application.added                        |2022-12-08 13:48:32.111 +0100|
project.application.config.api.added             |2022-12-08 13:48:32.111 +0100|
project.application.added                        |2022-12-08 13:48:32.111 +0100|
project.application.config.api.added             |2022-12-08 13:48:32.111 +0100|
project.application.added                        |2022-12-08 13:48:32.111 +0100|
project.application.config.oidc.added            |2022-12-08 13:48:32.111 +0100|
project.application.config.oidc.changed          |2022-12-08 13:48:32.111 +0100|
project.application.config.oidc.changed          |2022-12-08 13:48:32.111 +0100|
org.added                                        |2022-12-08 13:48:32.111 +0100|
org.domain.added                                 |2022-12-08 13:48:32.111 +0100|
org.domain.verified                              |2022-12-08 13:48:32.111 +0100|
org.domain.primary.set                           |2022-12-08 13:48:32.111 +0100|
org.member.added                                 |2022-12-08 13:48:32.111 +0100|
org.customtext.set                               |2022-12-08 14:54:57.314 +0100|
org.customtext.set                               |2022-12-08 14:54:57.314 +0100|
org.customtext.set                               |2022-12-08 14:54:57.314 +0100|
org.customtext.set                               |2022-12-08 14:54:57.314 +0100|
org.customtext.set                               |2022-12-08 14:54:57.314 +0100|
org.customtext.set                               |2022-12-08 14:54:57.314 +0100|
org.customtext.set                               |2022-12-08 14:55:10.575 +0100|
org.customtext.set                               |2022-12-08 14:55:10.575 +0100|
org.customtext.set                               |2022-12-08 14:55:10.575 +0100|
org.customtext.set                               |2022-12-08 14:55:10.575 +0100|
org.customtext.set                               |2022-12-08 14:55:10.575 +0100|
org.customtext.set                               |2022-12-08 14:55:10.575 +0100|
org.customtext.set                               |2022-12-08 14:57:29.291 +0100|
org.customtext.set                               |2022-12-08 14:57:42.789 +0100|
org.customtext.set                               |2022-12-17 01:45:17.525 +0100|
org.policy.label.added                           |2022-12-17 05:50:24.194 +0100|
org.policy.label.changed                         |2022-12-17 05:50:38.154 +0100|
org.policy.label.changed                         |2022-12-17 05:50:54.992 +0100|
org.policy.label.activated                       |2022-12-17 05:50:59.581 +0100|
org.policy.login.added                           |2022-12-22 13:03:10.131 +0100|
org.policy.login.secondfactor.added              |2022-12-22 13:03:10.131 +0100|
org.policy.login.secondfactor.added              |2022-12-22 13:03:10.131 +0100|
org.policy.login.multifactor.added               |2022-12-22 13:03:10.131 +0100|
org.changed                                      |2024-01-02 21:37:53.771 +0100|
org.domain.added                                 |2024-01-02 21:37:53.771 +0100|
org.domain.verified                              |2024-01-02 21:37:53.771 +0100|
org.domain.primary.set                           |2024-01-02 21:37:53.771 +0100|
org.domain.removed                               |2024-01-02 21:37:53.771 +0100|
org.policy.login.changed                         |2024-01-02 21:46:39.648 +0100|
org.policy.login.changed                         |2024-01-02 21:46:45.728 +0100|
org.domain.added                                 |2024-01-02 21:47:11.386 +0100|
org.domain.verified                              |2024-01-02 21:47:11.386 +0100|
instance.added                                   |2022-12-08 13:48:32.111 +0100|
instance.default.language.set                    |2022-12-08 13:48:32.111 +0100|
instance.secret.generator.added                  |2022-12-08 13:48:32.111 +0100|
instance.secret.generator.added                  |2022-12-08 13:48:32.111 +0100|
instance.secret.generator.added                  |2022-12-08 13:48:32.111 +0100|
instance.secret.generator.added                  |2022-12-08 13:48:32.111 +0100|
instance.secret.generator.added                  |2022-12-08 13:48:32.111 +0100|
instance.secret.generator.added                  |2022-12-08 13:48:32.111 +0100|
instance.secret.generator.added                  |2022-12-08 13:48:32.111 +0100|
instance.policy.password.complexity.added        |2022-12-08 13:48:32.111 +0100|
instance.policy.password.age.added               |2022-12-08 13:48:32.111 +0100|
instance.policy.domain.added                     |2022-12-08 13:48:32.111 +0100|
instance.policy.login.added                      |2022-12-08 13:48:32.111 +0100|
instance.policy.login.secondfactor.added         |2022-12-08 13:48:32.111 +0100|
instance.policy.login.secondfactor.added         |2022-12-08 13:48:32.111 +0100|
instance.policy.login.multifactor.added          |2022-12-08 13:48:32.111 +0100|
instance.policy.privacy.added                    |2022-12-08 13:48:32.111 +0100|
instance.policy.lockout.added                    |2022-12-08 13:48:32.111 +0100|
instance.policy.label.added                      |2022-12-08 13:48:32.111 +0100|
instance.policy.label.activated                  |2022-12-08 13:48:32.111 +0100|
instance.mail.template.added                     |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.customtext.set                          |2022-12-08 13:48:32.111 +0100|
instance.default.org.set                         |2022-12-08 13:48:32.111 +0100|
instance.member.added                            |2022-12-08 13:48:32.111 +0100|
instance.iam.project.set                         |2022-12-08 13:48:32.111 +0100|
instance.iam.console.set                         |2022-12-08 13:48:32.111 +0100|
instance.domain.added                            |2022-12-08 13:48:32.111 +0100|
instance.domain.primary.set                      |2022-12-08 13:48:32.111 +0100|
instance.domain.added                            |2022-12-08 13:48:32.111 +0100|
instance.domain.primary.set                      |2022-12-08 13:48:32.111 +0100|
instance.oidc.settings.added                     |2022-12-08 13:48:32.111 +0100|
instance.policy.domain.changed                   |2022-12-08 14:08:38.214 +0100|
instance.smtp.config.added                       |2022-12-08 14:11:57.576 +0100|
instance.smtp.config.password.changed            |2022-12-08 14:12:11.069 +0100|
instance.policy.label.changed                    |2022-12-08 14:12:47.488 +0100|
instance.policy.label.changed                    |2022-12-08 14:12:51.955 +0100|
instance.policy.label.icon.added                 |2022-12-08 14:13:29.460 +0100|
instance.policy.label.logo.added                 |2022-12-08 14:13:43.282 +0100|
instance.policy.label.changed                    |2022-12-08 14:13:57.092 +0100|
instance.policy.label.activated                  |2022-12-08 14:14:02.348 +0100|
instance.policy.label.icon.dark.added            |2022-12-08 14:14:10.069 +0100|
instance.policy.label.logo.dark.added            |2022-12-08 14:14:34.100 +0100|
instance.policy.label.changed                    |2022-12-08 14:14:43.268 +0100|
instance.policy.label.changed                    |2022-12-08 14:14:47.808 +0100|
instance.policy.label.changed                    |2022-12-08 14:14:58.783 +0100|
instance.policy.label.changed                    |2022-12-08 14:15:08.496 +0100|
instance.policy.label.changed                    |2022-12-08 14:15:12.643 +0100|
instance.policy.label.activated                  |2022-12-08 14:15:20.864 +0100|
instance.customtext.set                          |2022-12-08 14:15:37.824 +0100|
instance.customtext.set                          |2022-12-08 14:15:47.511 +0100|
instance.customtext.set                          |2022-12-08 14:15:55.856 +0100|
instance.customtext.set                          |2022-12-08 14:15:55.856 +0100|
instance.customtext.set                          |2022-12-08 14:15:55.856 +0100|
instance.customtext.set                          |2022-12-08 14:15:55.856 +0100|
instance.customtext.set                          |2022-12-08 14:15:55.856 +0100|
instance.customtext.set                          |2022-12-08 14:15:55.856 +0100|
instance.customtext.set                          |2022-12-08 14:16:05.433 +0100|
instance.customtext.set                          |2022-12-08 14:16:12.729 +0100|
instance.customtext.set                          |2022-12-08 14:16:20.174 +0100|
instance.customtext.set                          |2022-12-08 14:17:36.079 +0100|
instance.customtext.set                          |2022-12-08 14:17:43.810 +0100|
instance.customtext.set                          |2022-12-08 14:17:51.263 +0100|
instance.customtext.set                          |2022-12-08 14:17:51.263 +0100|
instance.customtext.set                          |2022-12-08 14:17:51.263 +0100|
instance.customtext.set                          |2022-12-08 14:17:51.263 +0100|
instance.customtext.set                          |2022-12-08 14:17:51.263 +0100|
instance.customtext.set                          |2022-12-08 14:17:51.263 +0100|
instance.customtext.set                          |2022-12-08 14:17:59.405 +0100|
instance.customtext.set                          |2022-12-08 14:18:11.571 +0100|
instance.customtext.set                          |2022-12-08 14:18:20.047 +0100|
instance.policy.privacy.changed                  |2022-12-08 14:18:55.797 +0100|
instance.customtext.set                          |2022-12-08 14:23:11.118 +0100|
instance.customtext.set                          |2022-12-17 01:55:49.861 +0100|
instance.customtext.removed                      |2022-12-17 01:55:49.861 +0100|
instance.customtext.set                          |2022-12-17 01:55:49.861 +0100|
instance.customtext.set                          |2022-12-17 01:56:17.761 +0100|
instance.customtext.removed                      |2022-12-17 01:56:17.761 +0100|
instance.customtext.set                          |2022-12-17 01:56:17.761 +0100|
instance.customtext.set                          |2022-12-17 01:56:33.785 +0100|
instance.customtext.set                          |2022-12-17 01:56:33.785 +0100|
instance.policy.label.changed                    |2022-12-17 05:40:16.965 +0100|
instance.policy.label.changed                    |2022-12-17 05:40:25.516 +0100|
instance.policy.label.changed                    |2022-12-17 05:40:55.225 +0100|
instance.policy.label.changed                    |2022-12-17 05:41:01.434 +0100|
instance.policy.label.changed                    |2022-12-17 05:42:28.307 +0100|
instance.policy.label.changed                    |2022-12-17 05:51:57.020 +0100|
instance.policy.label.activated                  |2022-12-17 05:51:59.809 +0100|
instance.policy.label.changed                    |2022-12-17 05:52:13.258 +0100|
instance.policy.label.changed                    |2022-12-17 05:52:20.247 +0100|
instance.policy.label.changed                    |2022-12-17 05:52:24.964 +0100|
instance.policy.label.activated                  |2022-12-17 05:52:26.694 +0100|
iam.idp.config.added                             |2022-12-19 00:31:58.490 +0100|
iam.idp.oidc.config.added                        |2022-12-19 00:31:58.490 +0100|
iam.idp.config.changed                           |2022-12-19 00:32:21.634 +0100|
instance.policy.login.idpprovider.added          |2022-12-19 00:32:43.658 +0100|
instance.policy.login.idpprovider.removed        |2022-12-19 01:08:08.870 +0100|
iam.idp.oidc.config.changed                      |2022-12-19 01:09:40.145 +0100|
instance.policy.login.idpprovider.added          |2022-12-19 01:09:52.228 +0100|
iam.idp.config.added                             |2022-12-19 01:30:29.215 +0100|
iam.idp.oidc.config.added                        |2022-12-19 01:30:29.215 +0100|
instance.policy.login.idpprovider.added          |2022-12-19 01:30:34.590 +0100|
iam.idp.oidc.config.changed                      |2022-12-19 01:31:45.237 +0100|
instance.policy.login.idpprovider.removed        |2022-12-19 01:32:03.730 +0100|
iam.idp.config.removed                           |2022-12-19 01:32:14.809 +0100|
instance.policy.login.idpprovider.cascade.removed|2022-12-19 01:32:14.809 +0100|
iam.idp.config.added                             |2022-12-19 01:43:22.117 +0100|
iam.idp.oidc.config.added                        |2022-12-19 01:43:22.117 +0100|
instance.policy.login.idpprovider.added          |2022-12-19 01:43:27.578 +0100|
iam.idp.oidc.config.changed                      |2022-12-19 01:49:31.508 +0100|
iam.idp.config.deactivated                       |2022-12-19 01:52:20.793 +0100|
instance.policy.login.idpprovider.removed        |2022-12-19 01:54:30.822 +0100|
iam.idp.config.added                             |2022-12-19 02:06:37.831 +0100|
iam.idp.oidc.config.added                        |2022-12-19 02:06:37.831 +0100|
instance.policy.login.idpprovider.added          |2022-12-19 02:06:42.864 +0100|
instance.policy.login.idpprovider.removed        |2022-12-19 02:07:27.655 +0100|
iam.idp.config.deactivated                       |2022-12-19 02:47:10.121 +0100|
iam.idp.config.added                             |2022-12-22 12:49:06.553 +0100|
iam.idp.oidc.config.added                        |2022-12-22 12:49:06.553 +0100|
instance.policy.login.idpprovider.added          |2022-12-22 12:49:12.771 +0100|
iam.idp.config.changed                           |2022-12-22 12:50:26.815 +0100|
iam.idp.config.changed                           |2022-12-22 12:50:49.152 +0100|
iam.idp.config.deactivated                       |2022-12-22 13:03:17.629 +0100|
instance.policy.login.idpprovider.removed        |2022-12-22 13:03:31.056 +0100|
instance.idp.gitlab.added                        |2023-03-18 07:36:34.256 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 07:36:49.348 +0100|
iam.idp.config.removed                           |2023-03-18 07:39:34.022 +0100|
instance.policy.login.idpprovider.cascade.removed|2023-03-18 07:39:34.022 +0100|
instance.idp.github.added                        |2023-03-18 07:47:08.733 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 07:47:15.101 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 07:49:49.269 +0100|
instance.idp.azure.added                         |2023-03-18 08:10:38.041 +0100|
instance.idp.azure.changed                       |2023-03-18 08:11:24.906 +0100|
instance.idp.oidc.changed                        |2023-03-18 08:11:43.170 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 08:11:48.047 +0100|
instance.idp.azure.changed                       |2023-03-18 08:13:15.423 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 08:22:11.763 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 08:30:18.913 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 08:32:48.633 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 08:37:40.076 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 08:39:02.136 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 08:42:16.849 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 08:42:38.316 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 08:47:17.845 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 08:48:47.324 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 08:59:57.525 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 09:00:16.935 +0100|
instance.idp.azure.changed                       |2023-03-18 09:08:46.302 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 09:08:51.212 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 09:10:29.820 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 09:11:07.240 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 09:12:50.303 +0100|
instance.idp.azure.changed                       |2023-03-18 09:48:48.278 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 10:05:35.741 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 10:07:54.832 +0100|
instance.idp.oidc.added                          |2023-03-18 10:39:06.035 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 10:39:17.921 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 10:41:52.949 +0100|
instance.idp.azure.changed                       |2023-03-18 10:42:47.309 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 10:43:07.897 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 10:52:44.524 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 11:06:07.655 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 11:06:53.728 +0100|
instance.policy.login.idpprovider.added          |2023-03-18 11:07:38.355 +0100|
instance.policy.login.idpprovider.removed        |2023-03-18 11:08:06.981 +0100|
instance.policy.login.idpprovider.added          |2023-04-14 22:36:09.685 +0200|
instance.idp.github.changed                      |2023-04-14 22:37:09.274 +0200|
instance.policy.login.idpprovider.added          |2023-04-14 22:40:21.747 +0200|
instance.policy.login.idpprovider.removed        |2023-04-14 22:42:28.763 +0200|
instance.idp.azure.changed                       |2023-04-14 22:51:11.502 +0200|
instance.idp.removed                             |2023-04-14 22:53:14.920 +0200|
instance.policy.login.idpprovider.added          |2023-04-14 22:54:59.931 +0200|
instance.idp.azure.changed                       |2023-04-14 22:57:47.635 +0200|
instance.idp.oidc.changed                        |2023-04-14 23:00:22.484 +0200|
instance.policy.login.idpprovider.added          |2023-04-14 23:00:42.673 +0200|
instance.policy.login.idpprovider.added          |2023-04-14 23:08:00.826 +0200|
instance.policy.login.idpprovider.removed        |2023-04-14 23:08:52.754 +0200|
instance.idp.oidc.changed                        |2024-01-02 21:39:13.602 +0100|
instance.idp.oidc.changed                        |2024-01-02 21:39:40.952 +0100|
instance.idp.oidc.changed                        |2024-01-02 21:40:02.646 +0100|

Jan 29 '24 15:01 CMiksche

FYI: I tried multiple versions now. From v2.43.6 and v2.42.11 I get the following error message:

ERROR request error oidc_error.parent="ID=QUERY-wu6Ee Message=Errors.App.NotFound Parent=(sql: no rows in result set)" oidc_error.description="unable to retrieve client by id" oidc_error.type=server_error

Version v2.41.14 seems to work. So the change which causes the Issue has to be between v2.41.14 and v2.42.11

The only thing which is different is the binary, database is same with updated migrations from 2.44

Jan 31 '24 12:01 CMiksche

Ok thanks for the info. That might help narrow down the problem. We did some fixes on 2.42 (2.42.13) and 2.43 (2.43.7) for changes done after 2.41, that might resolve that issue, but let me check.

Feb 01 '24 09:02 livio-a

2.42 (2.42.13) and 2.43 (2.43.7)

I tried these and also the most current version, 2.45.0. The migrations run without error but I still get the exact same error message ("Errors.App.NotFound")

Feb 14 '24 21:02 CMiksche

I created a Test Instance and found the concrete Issue!

The migrations don't work correctly. In my PostgreSQL Database new tables like apps6, instance_members4 and projects4 were created but the data from the previous tables - like apps5 and projects5 were not transferred.

Some tables like users seem to work but others - like the mentioned ones not.

What is irritating is that the migration step didn't output any error message. You should definitely improve the error handling because technical savvy users like me could manually fix the problem that way.

Feb 15 '24 10:02 CMiksche

Hello there,

i'm stuck on the same error and because of this on version 2.41.15. Upgrading to 2.48.2 leads to the following error:

grafik

together with these log entries:

time="2024-03-25T21:05:54Z" level=info msg="server is listening on [::]:8080" caller="/home/runner/work/zitadel/zitadel/cmd/start/start.go:508"
time="2024-03-25T21:06:05Z" level=error caller="/home/runner/work/zitadel/zitadel/internal/api/ui/login/renderer.go:341" error="ID=EVENT-adk13 Message=Errors.AuthRequest.UserAgentNotCorresponding"
time="2024-03-25T21:06:05Z" level=warning msg="instance by host" caller="/home/runner/work/zitadel/zitadel/internal/query/instance.go:210" domain=login.jufri.de error="failed to connect to `host=service-database-psql user=zitadel_database_user database=zitadel_database`: dial error (dial tcp 10.42.0.143:5432: operation was canceled)" host=login.jufri.de
time=2024-03-25T21:06:10.576Z level=WARN msg="request error" oidc_error.parent="ID=QUERY-wu6Ee Message=Errors.App.NotFound Parent=(sql: no rows in result set)" oidc_error.description=Errors.App.NotFound oidc_error.type=invalid_request status_code=400

I've also tried to go to Version 2.42.16, 2.43.10, 2.44.6, 2.45.4, 2.46.4 and 2.47.7 which are resulting in the same error. Migrations are running smooth.

Mar 25 '24 21:03 kiguigui

i'm stuck on the same error

FYI: You can probably solve this Issue by manually migrating the tables if you don't want to wait till the Zitadel Team has figured this problem out.

I created an article about it here: https://blog.m5e.de/post/manually-migrate-zitadel-with-postgresql/

Mar 26 '24 10:03 CMiksche

Hey Christoph,

thank you for you suggestion. I might consider that!

Mar 26 '24 20:03 kiguigui

@adlerhurst do you have an update on this problem?

Apr 01 '24 18:04 kiguigui

I have the same problem here.

All the time the migrations are failing or the same migration already started, will check again in 5 seconds error.

It's even funny, because even with a zero database - without Zitadel, without any migrations - it still gives the error.

So Zitadel is practically unusable. I can't even run it on a clean system, much less with already populated data.

Jun 13 '24 18:06 carlosmfreitas2409

I have the same problem here.

Do you try to upgrade from a version below 2.47.7? I only encountered this problem with older versions. Then I manually upgraded the Database (see my guide https://blog.m5e.de/post/manually-migrate-zitadel-with-postgresql/) and didn't have this problem afterwards.

I could theoretically close this Issue as my problem is solved, but I didn't do it so far because I would wish that the Zitadel Team provides some fix for us so that people don't have to fix their problem manually...

Jun 14 '24 13:06 CMiksche

zitadel zitadel copied to clipboard

[Bug]: --init-projections doesn't seem to work

Preflight Checklist

Environment

Version

Database

Database Version

Describe the problem caused by this bug

To reproduce

Screenshots

Expected behavior

Operating System

Relevant Configuration

Additional Context

zitadel
zitadel copied to clipboard