ERROR: relation "augur_operations.config" does not exist

[MoralCode]

I saw this once fairly recently and I suspect it is a current issue preventing new instances/DBs being created on the main branch

It seems to happen at the point when augur attempts to read its first config value from the DB

Stack Trace

psycopg2.errors.UndefinedTable: relation "augur_operations.config" does not exist
| init with redis redis://redis:6379
augur-1         | error: could not lock config file /augur/.gitconfig: Permission denied
augur-db-1      | 2025-11-18 21:09:39.579 UTC [5122] ERROR:  relation "augur_operations.config" does not exist at character 353
[SQL: SELECT augur_operations.config.id AS augur_operations_config_id, augur_operations.config.section_name AS augur_operations_config_section_name, augur_operations.config.setting_name AS augur_operations_config_setting_name, augur_operations.config.value AS augur_operations_config_value, augur_operations.config.type AS augur_operations_config_type
augur-1         | FROM augur_operations.config
augur-1         | WHERE augur_operations.config.section_name = %(section_name_1)s AND augur_operations.config.setting_name = %(setting_name_1)s]
augur-1         | [parameters: {'section_name_1': 'Redis', 'setting_name_1': 'cache_group'}]

Can anyone confirm if this is the case or help provide repro steps so I can validate this and fix it?

it is a release blocker if this is indeed an error

Previous slack messages from 2023 say:

That error indicates the schema didn’t get installed correctly. This can happen when one types the password in incorrectly. Try deleting the db.config.json file and doing make install again.

so maybe this is just transient or a one off, although it sounds to me like it could happen if password provided to augur on first startup doesnt match a password postgres accepts, causing augur to keep running even after trying and failing to initialize a DB, suggesting more robust augur setup that refuses to run if a valid schema doesnt exist at startup may be needed

[MoralCode]

@sgoggins does this match the issues you have observed on bare metal main branch?

if not please file a new issue for it. i really want to debug that ASAP

[shlokgilda]

Hey, I took a look at this and I think I've traced the failure chain. Pretty sure it's a Docker startup race condition combined with some silent error handling that's masking the real problem.

Here's what I think is happening:

1. docker compose up starts containers
2. augur container starts almost immediately after augur-db (since depends_on only waits for container start, not PostgreSQL readiness)
3. init.sh runs augur db create-schema while PostgreSQL is still initializing (which probably takes around 3-5s)
4. Alembic migration fails to connect... but subprocess.call() in db.py doesn't raise an exception, it just returns the exit code which gets ignored
5. Script continues, starts the app
6. App tries to read config -> table doesn't exist -> crash

[shlokgilda]

To add on to the previous comment, I think there are several places that need to be fixed:

1. No DB readiness check in docker/backend/init.sh: The script just runs migrations immediately without checking if PostgreSQL is actually accepting connections.
2. Silent failure in augur/application/cli/db.py:

def create_schema():
    call(["alembic", "upgrade", "head"])  # uses call(), not check_call()

subprocess.call() returns the exit code but doesn't raise on failure - so the script happily continues even when Alembic fails. 3. No graceful fallback in config layer: AugurConfig unconditionally adds DatabaseConfig as a source without checking if the table exists first.

[shlokgilda]

Side note on the .gitconfig error: The could not lock config file /augur/.gitconfig: Permission denied error in the logs looks like a separate issue - looks like a container permissions thing. Probably worth fixing but I don't think it's causing the config table crash.

[shlokgilda]

I think we can test this by starting with a clean slate. docker compose down -v and then try docker compose up --build again to see if we can recreate the error in the logs.

[MoralCode]

Side note on the .gitconfig error: The could not lock config file /augur/.gitconfig: Permission denied error in the logs looks like a separate issue - looks like a container permissions thing. Probably worth fixing but I don't think it's not causing the config table crash.

yeah this is likely coming from podmans differing permissions model compared to docker. Thats probably better for a different issue (i filed one a few weeks back for some podman/docker things but forget the exact number)

[MoralCode]

No graceful fallback in config layer AugurConfig unconditionally adds DatabaseConfig as a source without checking if the table exists first.

I think this is fine to allow the config layer to assume database access. i dont want to get too far into application execution without a database

[sgoggins]

I'm seeing the schema checks fail on bare metal in main, so this is likely related to that issue.

[MoralCode]

I'm seeing the schema checks fail on bare metal in main, so this is likely related to that issue.

Can you please share the logs in a new issue so i can more easily debug what happened?