immich icon indicating copy to clipboard operation
immich copied to clipboard

Unable to start immich_postgres on arm with 16k page size (rpi5 64bit)

Open mark-goodall opened this issue 5 months ago • 1 comments

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

  • [x] Yes

The bug

The issue appears to be jemalloc isn't configured to support 16K pagesize, which is what the kernel is configured as. This appears to be a known issue with jemalloc https://github.com/raspberrypi/bookworm-feedback/issues/107 https://github.com/jemalloc/jemalloc/issues/2639

PostgreSQL Database directory appears to contain a database; Skipping initialization

2025-06-09 08:57:14.156 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
2025-06-09 08:57:14.157 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 16 bytes failed
Using SSD storage

PostgreSQL Database directory appears to contain a database; Skipping initialization

2025-06-09 08:58:14.517 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
2025-06-09 08:58:14.517 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 16 bytes failed
Using SSD storage

This causes immich_server to restart continuously too:

Error: connect ETIMEDOUT
    at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
    at Object.onceWrapper (node:events:632:28)
    at Socket.emit (node:events:518:28)
    at Socket._onTimeout (node:net:609:8)
    at listOnTimeout (node:internal/timers:594:17)
    at process.processTimers (node:internal/timers:529:7) {
  errorno: 'ETIMEDOUT',
  code: 'ETIMEDOUT',
  syscall: 'connect'
}
microservices worker error: Error: write CONNECT_TIMEOUT database:5432, stack: Error: write CONNECT_TIMEOUT database:5432
    at connectTimedOut (/usr/src/app/node_modules/postgres/cjs/src/connection.js:257:20)
    at Timeout.done [as _onTimeout] (/usr/src/app/node_modules/postgres/cjs/src/connection.js:1035:8)
    at listOnTimeout (node:internal/timers:596:11)
    at process.processTimers (node:internal/timers:529:7)
/usr/src/app/node_modules/postgres/cjs/src/connection.js:388
    if (!query || typeof query !== 'object' || !query.reject) throw err
                                                              ^

Error: write CONNECT_TIMEOUT database:5432
    at connectTimedOut (/usr/src/app/node_modules/postgres/cjs/src/connection.js:257:20)
    at Timeout.done [as _onTimeout] (/usr/src/app/node_modules/postgres/cjs/src/connection.js:1035:8)
    at listOnTimeout (node:internal/timers:596:11)
    at process.processTimers (node:internal/timers:529:7) {
  code: 'CONNECT_TIMEOUT',
  errno: 'CONNECT_TIMEOUT',
  address: 'database',
  port: 5432
}

Node.js v22.14.0
microservices worker exited with code 1
Killing api process
Initializing Immich v1.134.0
Detected CPU Cores: 4
Starting api worker
Starting microservices worker
[Nest] 7  - 06/09/2025, 9:15:50 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 17  - 06/09/2025, 9:15:51 AM     LOG [Api:EventRepository] Initialized websocket server
microservices worker error: Error: getaddrinfo ENOTFOUND database, stack: Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
microservices worker exited with code 1
Killing api process
Initializing Immich v1.134.0
Detected CPU Cores: 4
Starting api worker
Starting microservices worker
[Nest] 7  - 06/09/2025, 9:16:01 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 17  - 06/09/2025, 9:16:02 AM     LOG [Api:EventRepository] Initialized websocket server
microservices worker error: Error: getaddrinfo ENOTFOUND database, stack: Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
microservices worker exited with code 1
Killing api process
Initializing Immich v1.134.0
Detected CPU Cores: 4
Starting api worker
Starting microservices worker
[Nest] 7  - 06/09/2025, 9:16:12 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 17  - 06/09/2025, 9:16:12 AM     LOG [Api:EventRepository] Initialized websocket server
microservices worker error: Error: getaddrinfo ENOTFOUND database, stack: Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
microservices worker exited with code 1
Killing api process
Initializing Immich v1.134.0
Detected CPU Cores: 4
Starting api worker
Starting microservices worker
[Nest] 7  - 06/09/2025, 9:16:22 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 17  - 06/09/2025, 9:16:23 AM     LOG [Api:EventRepository] Initialized websocket server
microservices worker error: Error: getaddrinfo ENOTFOUND database, stack: Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
microservices worker exited with code 1
Killing api process
Initializing Immich v1.134.0
Detected CPU Cores: 4
Starting api worker
Starting microservices worker
[Nest] 7  - 06/09/2025, 9:16:34 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 17  - 06/09/2025, 9:16:34 AM     LOG [Api:EventRepository] Initialized websocket server
microservices worker error: Error: getaddrinfo ENOTFOUND database, stack: Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
microservices worker exited with code 1
Killing api process
Initializing Immich v1.134.0
Detected CPU Cores: 4
Starting api worker
Starting microservices worker
[Nest] 7  - 06/09/2025, 9:16:46 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 17  - 06/09/2025, 9:16:47 AM     LOG [Api:EventRepository] Initialized websocket server

I will switch to a different kernel for now (This prevents the postgres container from crashing) and it looks like PR upstream are in process to fix the issue, I'm not sure how quickly this will trickle down into postgres. I can't see a dockerfile for the postgres container...

The OS that Immich Server is running on

Raspbian OS on RPi5, 64bit with 16K pagesize

Version of Immich Server

v1.134

Version of Immich Mobile App

latest

Platform with the issue

  • [x] Server
  • [ ] Web
  • [ ] Mobile

Your docker-compose.yml content

default

Your .env content

default

Reproduction steps

  1. docker compose up -d
  2. docker ps # see restarting containers
  3. docker logs # see logs ...

Relevant log output


Additional information

No response

mark-goodall avatar Jun 09 '25 09:06 mark-goodall

Having the same problem.

mikhasd avatar Jun 13 '25 22:06 mikhasd

I'm getting this (on RPI5, Debian/Bookworm) with the postgres image referenced in the v1.135.0 docker-compose.yml: ghcr.io/immich-app/postgres:14-vectorchord0.4.1-pgvectors0.2.0

immich_postgres          | <jemalloc>: Unsupported system page size
immich_postgres          | memory allocation of 16 bytes failed
immich_postgres exited with code 133

Reverting back to ghcr.io/immich-app/postgres:14-vectorchord0.3.0-pgvectors0.2.0 works.

t-paul avatar Jun 18 '25 22:06 t-paul

I'm getting this (on RPI5, Debian/Bookworm) with the postgres image referenced in the v1.135.0 docker-compose.yml: ghcr.io/immich-app/postgres:14-vectorchord0.4.1-pgvectors0.2.0

immich_postgres          | <jemalloc>: Unsupported system page size
immich_postgres          | memory allocation of 16 bytes failed
immich_postgres exited with code 133

Reverting back to ghcr.io/immich-app/postgres:14-vectorchord0.3.0-pgvectors0.2.0 works.

Same issue here. Using HDD storage mode as my immich data is all stored on an external WD 4TB drive.

geekykid12 avatar Jun 19 '25 03:06 geekykid12

Got the same issue, as others already stated, simply reverting back to 0.3.0 works fine for now.

jasonlessenich avatar Jun 19 '25 06:06 jasonlessenich

Is the result any different if you remove -pgvectors0.2.0 from the image name and redeploy?

mertalev avatar Jun 19 '25 08:06 mertalev

Is the result any different if you remove -pgvectors0.2.0 from the image name and redeploy?

No, same result.

kosi2801 avatar Jun 19 '25 12:06 kosi2801

The helpful team at tensorchord has announced an updated version soon, that should fix this issue 😃

https://github.com/tensorchord/VectorChord/issues/277#issuecomment-2988461474

kosi2801 avatar Jun 19 '25 15:06 kosi2801

The updated VectorChord version 0.4.3 that should fix the issues with rpi5 has just been released.

kosi2801 avatar Jun 20 '25 07:06 kosi2801

Yep, confirmed, ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0 fixes the issue on RPI5. Thank you!

t-paul avatar Jun 20 '25 16:06 t-paul

Yep, confirmed, ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0 fixes the issue on RPI5. Thank you!

How long did it take for it to boot back up after the upgrade? My immich_server is reporting unhealthy

geekykid12 avatar Jun 20 '25 20:06 geekykid12

How long did it take for it to boot back up after the upgrade? My immich_server is reporting unhealthy

For me just a couple of seconds, but that was just from the previous version. If you upgrade from < 1.133 make sure to check those release notes: https://github.com/immich-app/immich/releases/tag/v1.133.0

t-paul avatar Jun 20 '25 21:06 t-paul

How long did it take for it to boot back up after the upgrade? My immich_server is reporting unhealthy

About 20-30s for me. Can you find any error in the logs of the immich postgres or immich server?

kosi2801 avatar Jun 20 '25 21:06 kosi2801

Looks like I am at the limit of diskspace for postgres to function properly, stats collector isn't functioning and immich_server is reporting unhealthy status. SD card has 1.2GB available after as much pruning as I can do. At the point now where I can't pull down 1.135.2 to see if that fixes my issue.

geekykid12 avatar Jun 21 '25 02:06 geekykid12

Diskspace was my issue, moved my docker data-root to an external drive and issues are resolved.

geekykid12 avatar Jun 21 '25 05:06 geekykid12