after upgrade to 117.0 the postgres CrashLoopBackOff

[addich]

The bug

i had version v1.116.2 successful running.

then i did:

helm repo update vi values.yaml # change version from v1.116.2 to v1.117.0 helm upgrade --namespace immich immich immich/immich -f values.yaml

afterwards the postgres start to crashloop

$ kubectl -n immich logs immich-postgresql-0 chmod: changing permissions of '/var/run/postgresql': Read-only file system

PostgreSQL Database directory appears to contain a database; Skipping initialization

2024-10-03 21:19:09.780 UTC [1] LOG: starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit 2024-10-03 21:19:09.780 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 2024-10-03 21:19:09.780 UTC [1] LOG: listening on IPv6 address "::", port 5432 2024-10-03 21:19:09.789 UTC [1] FATAL: could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock": Read-only file system 2024-10-03 21:19:09.794 UTC [1] LOG: database system is shut down

downgrade to old version does not change anything

The OS that Immich Server is running on

kubernetes (k3s on ubuntu)

Version of Immich Server

v1.117.0

Version of Immich Mobile App

v1.116.1

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

HELM values files:

## This chart relies on the common library chart from bjw-s
## You can find it at https://github.com/bjw-s/helm-charts/tree/main/charts/library/common
## Refer there for more detail about the supported values

# These entries are shared between all the Immich components

env:
  REDIS_HOSTNAME: '{{ printf "%s-redis-master" .Release.Name }}'
  DB_HOSTNAME: "{{ .Release.Name }}-postgresql"
  DB_USERNAME: "{{ .Values.postgresql.global.postgresql.auth.username }}"
  DB_DATABASE_NAME: "{{ .Values.postgresql.global.postgresql.auth.database }}"
  # -- You should provide your own secret outside of this helm-chart and use `postgresql.global.postgresql.auth.existingSecret` to provide credentials to the postgresql instance
  DB_PASSWORD: "{{ .Values.postgresql.global.postgresql.auth.password }}"
  IMMICH_MACHINE_LEARNING_URL: '{{ printf "http://%s-machine-learning:3003" .Release.Name }}'

image:
  tag: v1.116.2

immich:
  metrics:
    # Enabling this will create the service monitors needed to monitor immich with the prometheus operator
    enabled: false
  persistence:
    # Main data store for all photos shared between different components.
    library:
      # Automatically creating the library volume is not supported by this chart
      # You have to specify an existing PVC to use
      existingClaim: immich-data
  # configuration is immich-config.json converted to yaml
  # ref: https://immich.app/docs/install/config-file/
  #
  configuration: {}
    # trash:
    #   enabled: false
    #   days: 30
    # storageTemplate:
    #   enabled: true
    #   template: "{{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}"

# Dependencies

postgresql:
  enabled: true
  image:
    repository: tensorchord/pgvecto-rs
    tag: pg14-v0.2.0
  resources:
    limits:
      cpu: "1"
  global:
    postgresql:
      auth:
        username: immich
        database: immich
        password: DELETED
  primary:
    initdb:
      scripts:
        create-extensions.sql: |
          CREATE EXTENSION cube;
          CREATE EXTENSION earthdistance;
          CREATE EXTENSION vectors;

redis:
  enabled: true
  architecture: standalone
  auth:
    enabled: false

# Immich components

server:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-server
    pullPolicy: IfNotPresent
  resources:
    limits:
      cpu: "2"
  ingress:
    main:
      enabled: true
      annotations:
        # proxy-body-size is set to 0 to remove the body limit on file uploads
        nginx.ingress.kubernetes.io/proxy-body-size: "0"
        nginx.ingress.kubernetes.io/proxy-connect-timeout: "120"
        nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
        nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
        nginx.ingress.kubernetes.io/proxy-next-upstream: "error timeout non_idempotent"
        cert-manager.io/cluster-issuer: "letsencrypt"
        nginx.ingress.kubernetes.io/modsecurity-snippet: |
          SecRule REQUEST_URI "@beginsWith /api/assets" "id:9001,phase:1,t:none,pass,nolog,ctl:requestBodyAccess=Off,ctl:ruleRemovebyID=1013,ctl:ruleRemovebyID=921110"
          SecAction "id:901004,phase:1,nolog,pass,t:none,setvar:tx.allowed_methods=GET HEAD POST OPTIONS DELETE PUT PATCH"
          SecAction "id:901005,phase:1,nolog,pass,t:none,setvar:tx.allowed_request_content_type=|application/x-www-form-urlencoded| |multipart/form-data| |multipart/related| |text/xml| |application/xml| |application/soap+xml| |application/json| |application/cloudevents+json| |application/cloudevents-batch+json| |text/plain|"
      hosts:
        - host: photos.addi.ch
          paths:
            - path: "/"
      ingressClassName: nginx
      tls:
        - hosts:
          - photos.addi.ch
          secretName: ingress-tls
  persistence:
    external:
      enabled: true
      readOnly: true
      type: nfs
      server: nfs
      path: /k8s/immich-external-library

machine-learning:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-machine-learning
    pullPolicy: IfNotPresent
  resources:
    limits:
      cpu: "500m"
  env:
    TRANSFORMERS_CACHE: /cache
  persistence:
    cache:
      enabled: true
      size: 10Gi
      # Optional: Set this to pvc to avoid downloading the ML models every start.
      type: emptyDir
      accessMode: ReadWriteMany
      # storageClass: your-class

Your .env content

NA

Reproduction steps

1. helm repo update
2. vi values.yaml # change version from v1.116.2 to v1.117.0
3. helm upgrade --namespace immich immich immich/immich -f values.yaml

Relevant log output

$ kubectl -n immich logs immich-postgresql-0
chmod: changing permissions of '/var/run/postgresql': Read-only file system

PostgreSQL Database directory appears to contain a database; Skipping initialization

2024-10-03 21:19:09.780 UTC [1] LOG:  starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-10-03 21:19:09.780 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-10-03 21:19:09.780 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-10-03 21:19:09.789 UTC [1] FATAL:  could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock": Read-only file system
2024-10-03 21:19:09.794 UTC [1] LOG:  database system is shut down

Additional information

No response