supabase
Auth service fails to start, can't create auth schema / can't execute migrations (when deployed via helm)
Bug report
- [x] I confirm this is a bug with Supabase, not with my own application.
- [x] I confirm I have searched the Docs, GitHub Discussions, and Discord.
Describe the bug
In short: no schema has been selected to create in error during running the migrations.
In long: Auth schema is not being created and migrations are failing to be executed when running supabase in kubernetes via helm (using official community helmchart).
I'm using as-default-as-possible settings; no tweaks are made to migrations, helm templates or anything like that.
The issue has been reported in helm's repository https://github.com/supabase-community/supabase-kubernetes/issues/86 but it seems that it's not a helm wrapper issue, and rather is related to an inconsistency in migration files + assumptions of who pre-creates auth schema, depending on how supabase and its services are being deployed.
Let's compare how differently supabase can be deployed:
Option 1. supabase CLI (all works well)
If deployed via supabase CLI (supabase init; supabase start, which spins docker-compose with several docker containers inside), then the following happens
-
dbcontainer creates schemas via the entry-point script which creates/alters some users (includingsupabase_auth_adminandpostgres) and some non-auth related schemas, tables, functions and extensions (I couldn't find the source of this code, but something very similar has been found in https://github.com/supabase/supabase/blob/fbef43fbbcefea1cbb03c8e932681a5ad4adbcbf/docker/volumes/db/roles.sql and sibling files):
"Entrypoint": [
"sh",
"-c",
"\ncat <<'EOF' > /etc/postgresql.schema.sql && \\\ncat <<'EOF' > /etc/postgresql-custom/pgsodium_root.key && \\\ncat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\ndocker-entrypoint.sh postgres -D /etc/postgresql -c search_path='$user,public,extensions'\n\\set pgpass `echo \"$PGPASSWORD\"`\n\\set jwt_secret `echo \"$JWT_SECRET\"`\n\\set jwt_exp `echo \"$JWT_EXP\"`\n\nALTER DATABASE postgres SET \"app.settings.jwt_secret\" TO :'jwt_secret';\nALTER DATABASE postgres SET \"app.settings.jwt_exp\" TO :'jwt_exp';\n\nALTER USER postgres WITH PASSWORD :'pgpass'..."
],
-
dbcontainer also runs other init scripts from https://github.com/supabase/postgres/blob/develop/migrations/db/init-scripts/00000000000001-auth-schema.sql, which createauthschema and assign it tosupabase_admin:
...
CREATE SCHEMA IF NOT EXISTS auth AUTHORIZATION supabase_admin;
...
-
authcontainer runs migrations from https://github.com/supabase/auth/blob/master/migrations/00_init_auth_schema.up.sql and other files. But those files are identical to the the ones that PG already executed, soauth's migrations cause a no-op:
it's already in up to date state
Option 2. supabase helm with supabase version of postgres
When running via helm, and keeping supabase's default PG instead of using RDS or other pre-existing PG, then here's what happens:
- there's no 1st step in
dbpod/container - no one creates asupabase_adminrole. - but
dbpod is trying to run the same scripts as in 2nd step (https://github.com/supabase/postgres/blob/develop/migrations/db/init-scripts/00000000000001-auth-schema.sql, etc):
echo "Copying init scripts into existing image script directory..."
cp -r /docker-entrypoint-initdb.d/* /initdb.d/
cp /custom-init-scripts/98-webhooks.sql /initdb.d/init-scripts/
...
which copies script files well, but fails to execute them:
92.168.135.3 2025-07-30 11:57:04.361 UTC [50] postgres@postgres ERROR: relation "schema_migrations" does not exist at character 15
192.168.135.3 2025-07-30 11:57:04.361 UTC [50] postgres@postgres STATEMENT: select * from schema_migrations
192.168.135.3 2025-07-30 11:57:04.363 UTC [50] postgres@postgres LOG: statement: CREATE TABLE "schema_migrations" (
"version" VARCHAR (14) NOT NULL,
PRIMARY KEY("version")
);
CREATE UNIQUE INDEX "schema_migrations_version_idx" ON "schema_migrations" (version);
192.168.135.3 2025-07-30 11:57:04.363 UTC [50] postgres@postgres ERROR: no schema has been selected to create in at character 14
192.168.135.3 2025-07-30 11:57:04.363 UTC [50] postgres@postgres STATEMENT: CREATE TABLE "schema_migrations" (
"version" VARCHAR (14) NOT NULL,
PRIMARY KEY("version")
);
CREATE UNIQUE INDEX "schema_migrations_version_idx" ON "schema_migrations" (version);
192.168.99.130 2025-07-30 11:57:05.209 UTC [52] postgres@postgres LOG: execute <unnamed>: CREATE TABLE IF NOT EXISTS "schema_migrations" ("version" bigint, "inserted_at" timestamp(0), PRIMARY KEY ("version"))
192.168.99.130 2025-07-30 11:57:05.209 UTC [52] postgres@postgres ERROR: no schema has been selected to create in at character 28
- and finally, when
authpod runs migrations, if fails with the same error:
{"level":"info","msg":"Go runtime metrics collection started","time":"2025-07-29T10:55:33Z"}
{"args":["20240115144230_remove_ip_address_from_saml_relay_state.sql"],"component":"pop","level":"warning","msg":"ignoring file %s because it does not match the migration file pattern","time":"2025-07-29T10:55:33Z"}
{"args":[0.015577543],"component":"pop","level":"info","msg":"%.4f seconds","time":"2025-07-29T10:55:33Z"}
{"level":"fatal","msg":"running db migrations: Migrator: problem creating schema migrations: could not execute CREATE TABLE \"schema_migrations\" (\n\"version\" VARCHAR (14) NOT NULL,\nPRIMARY KEY(\"version\")\n);\nCREATE UNIQUE INDEX \"schema_migrations_version_idx\" ON \"schema_migrations\" (version);: ERROR: no schema has been selected to create in (SQLSTATE 3F000)","time":"2025-07-29T10:55:33Z"}
Option 3. supabase helm with external postgres
this is the worst case in terms of inconsistency:
- external db doesn't know about any special roles to be created or init scripts to be executed. even worse - there may be no access to modify its run time scripts, but even if there were, then they would not be executed, since they typically run only at the DB volume creation time, but not after
- when
authservice runs migrations, fails with the same error
ERROR: no schema has been selected to create in (SQLSTATE 3F000)
Proposal for the fix
- remove hardcoded roles such as
supabase_adminfrom migrations and scripts, as it may not work well with external DBs. Parametrize it. - instead of hardcoded
supabase-cli's entrypoint script, use the script(s) from supabase/supabase repo, mounted as files and not hardcoded. - move all migrations out of supabase/postgres into respective services into a new migration service/job to remove duplication between PG-migrations and Auth-migrations, and to allow external DBs to be migrated as well
- split migration process into 2
- master migrations (which require master PG password): create roles, create schemas, do initial grants
- auth-specific migrations (which require non-master PG password): those are the regular auth-service migrations
- that will require passing 2 types of credentials to auth service or a migrator service (if this service will be created)
To Reproduce
(i couldn't attach a file, sorry)
Before the following steps, please first save that snippet as sb-values.yaml:
secret:
jwt:
anonKey: "test-test-test"
serviceKey: "test-test-test2"
secret: "test-test-test3"
db:
username: "postgres"
password: "postgres"
database: "postgres"
analytics:
apiKey: "test-test-test-test"
smtp:
username: "user"
password: "pass"
dashboard:
username: user
password: pass
db:
enabled: true
image:
repository: public.ecr.aws/supabase/postgres
tag: "15.8.1.085"
auth:
enabled: true
image:
repository: public.ecr.aws/supabase/gotrue
pullPolicy: IfNotPresent
tag: "v2.177.0"
environment:
DB_USER: postgres
# DB_NAMESPACE: auth
LOG_LEVEL: debug
GOTRUE_LOG_LEVEL: debug
GOTRUE_SMTP_ADMIN_EMAIL: "[email protected]"
GOTRUE_SMTP_HOST: smtp.gmail.com
GOTRUE_SMTP_PORT: 587
GOTRUE_SMTP_SENDER_NAME: test
rest:
enabled: true
image:
repository: public.ecr.aws/supabase/postgrest
tag: "v12.2.12"
environment:
DB_USER: postgres
studio:
enabled: true
realtime:
enabled: false
meta:
enabled: false
storage:
enabled: false
imgproxy:
enabled: false
kong:
enabled: true
image:
repository: kong
pullPolicy: IfNotPresent
tag: "latest"
analytics:
enabled: false
vector:
enabled: false
functions:
enabled: false
minio:
enabled: false
- install minukube locally, or connect to any other k8s cluster
- install helm (e.g.
brew install helm) - install supabase into the cluster:
helm upgrade --install supabase vastio/supabase --namespace supabase --create-namespace -f sb-values.yaml
- observe that Kong, DB, Studio and Rest services are up and running, but Auth is not (see screenshot)
- check the
Authpod's logs (see screenshot)
Expected behavior
Regardless of using of a SB-PG DB or a pre-existing external PG DB, all migrations should have been applied and Auth service should have been running green
Screenshots
When running in kubernetes - bad:
When running via supabase CLI - good:
System information
Irrelevant
Additional context
on top of non-tweaked configs, i've also tried tweaking them a bit (tried different variations, that's why there are many similar things - i didn't know which ones are taken into account):
DB_NAMESPACE: auth
LOG_LEVEL: debug
GOTRUE_LOG_LEVEL: debug
GOTRUE_DB_DRIVER: postgres
GOTRUE_DB_DATABASE_URL: postgresql://postgres:postgres@supabase-supabase-db:5432/postgres
DB_AFTER_CONNECT_QUERY: "SET search_path TO auth,public"
GOTRUE_DB_AFTER_CONNECT_QUERY: "SET search_path TO auth,public"
PGOPTIONS: -c search_path=auth,public
GOTRUE_PGOPTIONS: -c search_path=auth,public
GOTRUE_PG_OPTIONS: -c search_path=auth,public
GOTRUE_DB_PGOPTIONS: -c search_path=auth,public
GOTRUE_DB_PG_OPTIONS: -c search_path=auth,public
And that caused a migration process to progress a bit further along: it stopped looking for non-existant auth, and moved on to public, created migrations table there but stopped on some other issue later on.
But that confirms that the auth schema is expected to be pre-created before auth service could run its migrations, and it makes it hard to run supabase stack outside of tuned supabase cli environment.
