Could not start database server

[christoph-buente]

Hi @pellepelster,

thanks for putting so much effort into making the RDS system work on Hetzner. It's a great set of tools and Hetzner should consider adopting it. We want to start using it in a production environment and we are testing out a lot of scenarios to make sure we can be up and running with our project at all times.

Today we ran into an issue when rotating the db host. I admit it's an edge case, but wanted to report it anyway. Here's what we did:

Current setup

• We have a running RDS Host with a database, our app is able to connect
• We have a mounted data volume that has the DB data
• We have an S3 bucket in place containing full and incremental backups

What is the issue

We discovered, that the incremental backups caused a lot of HEAD requests to the S3 API, which exhausted our free quota in Backblaze. I configured the incremental backups to be done every 4 hours instead of every hour by setting this value in Terraform:

    backup_incr_calendar = "*-*-* 00/4:30"

This caused the cloud-init script to change, which in turn forced the RDS host to be rotated.

When the new host started, it found the PGDATA directory on the mounted data volume. It starts the postgres server and i can connect to it. However, the S3 API returned a 403 forbidden when it wanted to check the database backups on S3. This caused the container to stop, leaving the database server not running.

Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m 2024-02-19 14:03:08.328 P00   INFO: stanza-create for stanza 'database01' on repo1
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m ERROR: [039]: HTTP request failed with 403:
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        *** Path/Query ***:
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        HEAD /pgbackrest/archive/database01/archive.info
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        *** Request Headers ***:
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        authorization: <redacted>
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        content-length: 0
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        host: <redacted>.backblazeb2.com
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        x-amz-content-sha256: <redacted>
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        x-amz-date: <redacted>
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        *** Response Headers ***:
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        cache-control: max-age=0, no-cache, no-store
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        content-length: 259
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        content-type: application/xml
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        date: Mon, 19 Feb 2024 14:03:08 GMT
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        x-amz-id-2: <redacted>
Feb 19 14:03:08 database01 docker-compose[15574]: #033[36mdatabase01_postgresql |#033[0m        x-amz-request-id: <redacted>

What do I expect?

If I have a working Postgres database with data volume and the Postgres server came up successfully, I would appreciate a warning about the S3 backup not being reachable. But I would expect the Postgres server to stay running instead of the container to stop.

Appreciate your thoughts on this.