immich
immich copied to clipboard
[BUG] Docker stack can not connect to DB
The bug
For unknow reason my docker crashed.
After that i am unable to get Immich to work. I have tried backup of the whole VM running docker and also tried to create new networks and assign them in the compose file, they are all on the same network but booth immich_server and immich_microservices
get this:
[Nest] 1 - 05/19/2023, 3:42:12 AM ERROR [TypeOrmModule] Unable to connect to the database. Retrying (5)...
Error: Connection terminated due to connection timeout
at Connection.
Its just like they are on the same network but cant talk.
i have compared to the example compose yaml and env file, but nothing sticks out and this happend from one day to another. also when i restarted the VM this also killed the connection to portainer.
The OS that Immich Server is running on
Ubuntu Server
Version of Immich Server
several, v1.56.0
Version of Immich Mobile App
Platform with the issue
- [X] Server
- [X] Web
- [ ] Mobile
Your docker-compose.yml content
version: "3.8"
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:release
entrypoint: ["/bin/sh", "./start-server.sh"]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- stack.env
depends_on:
- redis
- database
- typesense
restart: always
networks:
- immich_prod
immich-microservices:
container_name: immich_microservices
image: ghcr.io/immich-app/immich-server:release
entrypoint: ["/bin/sh", "./start-microservices.sh"]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- stack.env
depends_on:
- redis
- database
- typesense
restart: always
networks:
- immich_prod
immich-machine-learning:
container_name: immich_machine_learning
image: ghcr.io/immich-app/immich-machine-learning:release
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- immich_prod_truenas_model-cache:/cache
env_file:
- stack.env
restart: always
networks:
- immich_prod
immich-web:
container_name: immich_web
image: ghcr.io/immich-app/immich-web:release
entrypoint: ["/bin/sh", "./entrypoint.sh"]
env_file:
- stack.env
restart: always
networks:
- immich_prod
typesense:
container_name: immich_typesense
image: typesense/typesense:0.24.0
environment:
- TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
- TYPESENSE_DATA_DIR=/data
logging:
driver: none
volumes:
- immich_prod_truenas_tsdata:/data
restart: always
networks:
- immich_prod
redis:
container_name: immich_redis
image: redis:6.2
restart: always
networks:
- immich_prod
database:
container_name: immich_postgres
image: postgres:14
env_file:
- stack.env
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
PG_DATA: /var/lib/postgresql/data
volumes:
- immich_prod_immich_prod_truenas_pgdata:/var/lib/postgresql/data
restart: always
networks:
- immich_prod
immich-proxy:
container_name: immich_proxy
image: ghcr.io/immich-app/immich-proxy:release
environment:
# Make sure these values get passed through from the env file
- IMMICH_SERVER_URL
- IMMICH_WEB_URL
ports:
- 2283:8080
logging:
driver: none
depends_on:
- immich-server
restart: always
networks:
- immich_prod
volumes:
immich_prod_immich_prod_truenas_pgdata:
external: false
immich_prod_truenas_model-cache:
external: true
immich_prod_truenas_tsdata:
external: true
immich_prod_truenas_upload:
external: true
networks:
immich_prod:
external: true
Your .env content
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=#SECRET#
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=immich_prod_truenas_upload
TYPESENSE_API_KEY=#SECRET#
PUBLIC_LOGIN_PAGE_MESSAGE=
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
Reproduction steps
Deploy stack
Additional information
immich_prod_immich_prod_truenas_pgdata: = Local external: false immich_prod_truenas_model-cache: = On CIFS external: true immich_prod_truenas_tsdata: = On CIFS external: true immich_prod_truenas_upload: = On CIFS external: true
Try to bring down the whole stack, then remove the database container. Then bring back the whole stack to recreate them
would that differ from recrating the whole stack as a clean slate several times? Im in the begining of transfering to immich, so i also removed all already uploaded pictures. so all volumes are clean
I suspect this has to do with the database container getting created. If you bring down the container and bring it up, it doesn't necessarily remove the container. Try to explicitly remove those container and then bring it up
Also, please try to remove the network config below
networks:
immich_prod:
external: true
removed everything with network back to stock. Now this failed for unclear reason and now i cant reach portainer anymore. Similar behavior as earlier. this feels like something isnt right with the networking when the stack gets deployed.
I have the same problem. immich_server and immich_microservices has this error:
Error: Connection terminated due to connection timeout
at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
at Object.onceWrapper (node:events:641:28)
at Connection.emit (node:events:527:28)
at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:57:12)
at Socket.emit (node:events:527:28)
at TCP.<anonymous> (node:net:709:12)
Followed instructions from https://immich.app/docs/install/portainer, no network changes.
Try to bring down the whole stack, then remove the database container. Then bring back the whole stack to recreate them
When I stopped the stack with Portainer, I verified that the database container (all containers, for that matter) gets removed and recreated.
@eltiel Your error is different, how much resources did you allocate for Immich?
@eltiel Your error is different, how much resources did you allocate for Immich?
I'm running on a Synology DS218+ - dual-core Intel Celeron J3355 with 10GB RAM (5GB free). As far as I know, I did not impose any memory limit.
I ran into similar problems when upgrading from 1.51 to 1.52 and above. The database migrations overwhelmed the database container and its CPU usage was at max, causing database connections to run into timeouts. The migrations took so long that the database container kept crashing and restarting and the migrations never finished. I was able to solve it by allocating more ram to the database and setting jit = off in the postgres config. Hope this works for you too.
@kriskbx It was actually my very first time installing Immich so there wasn't anything to migrate. I did check my CPU and memory usage and nothing was out of the ordinary, CPU certainly wasn't maxed out. My containers didn't crash though, it sort of just sat there.
Hi!
I will try the jit = off setting, maybe that is what is happening for me. I doesn’t explain the initial crash tho.
but does anyone have an idea on what happend to my docket during last try, I can’t reach my portainer, the symptom is like the standard bridge is not accessible when the setup failed on Immich.
Reboot does not solve it. Reinstall portainer does not solve it. Is there som setting that the docker compose for Immich might have disabled?
So i managed to revert back to an even older backup. So now im back on track. all of a sudden it worked to start the stack with out DB errors.
But still i cant access the web interface and i dont understand where i should put in the jit = off argument. Please help
@eltiel I have the same issue.
@alextran1502 I am running it on Dell R730XD server, with 18 cores CPU, 32G RAM. All the resources were not ran out. So I don't think it come from the limitation of the resources
@kriskbx It was actually my very first time installing Immich so there wasn't anything to migrate. I did check my CPU and memory usage and nothing was out of the ordinary, CPU certainly wasn't maxed out. My containers didn't crash though, it sort of just sat there.
I updated the Portianer.io to the latest version, deleted the whole Immich stack, and reinstall it. Then, it suddenly worked.
Hi, I'm facing a similar error immich server logs
[Nest] 1 - 05/20/2023, 6:45:14 PM ERROR [TypeOrmModule] Unable to connect to the database. Retrying (3)...
Error: connect ECONNREFUSED 127.0.0.1:5432
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
I am deploying via portainer.
Docker Compose
version: "3.8"
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:release
entrypoint: ["/bin/sh", "./start-server.sh"]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
environment:
DB_PASSWORD: ${DB_PASSWORD}
DB_USERNAME: ${DB_USERNAME}
DB_DATABASE_NAME: ${DB_DATABASE_NAME}
TYPESENSE_API_KEY: ${TYPESENSE_API_KEY}
depends_on:
- redis
- database
- typesense
restart: always
immich-microservices:
container_name: immich_microservices
image: ghcr.io/immich-app/immich-server:release
entrypoint: ["/bin/sh", "./start-microservices.sh"]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
depends_on:
- redis
- database
- typesense
restart: always
immich-machine-learning:
container_name: immich_machine_learning
image: ghcr.io/immich-app/immich-machine-learning:release
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- model-cache:/cache
restart: always
immich-web:
container_name: immich_web
image: ghcr.io/immich-app/immich-web:release
entrypoint: ["/bin/sh", "./entrypoint.sh"]
restart: always
typesense:
container_name: immich_typesense
image: typesense/typesense:0.24.0
environment:
- TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
- TYPESENSE_DATA_DIR=/data
logging:
driver: none
volumes:
- tsdata:/data
restart: always
redis:
container_name: immich_redis
image: redis:6.2
restart: always
database:
container_name: immich_postgres
image: postgres:14
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
PG_DATA: /var/lib/postgresql/data
volumes:
- /srv/dev-disk-by-uuid-6bcdc785-ff84-4338-824f-ac76e5d9695c/docker/immich/pgdata:/var/lib/postgresql/data
restart: always
immich-proxy:
container_name: immich_proxy
image: ghcr.io/immich-app/immich-proxy:release
environment:
# Make sure these values get passed through from the env file
- IMMICH_SERVER_URL= ${IMMICH_SERVER_URL}
- IMMICH_WEB_URL= ${IMMICH_WEB_URL}
ports:
- 1016:8080
logging:
driver: none
depends_on:
- immich-server
restart: always
volumes:
pgdata:
model-cache:
tsdata:
Environment variables
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=<redacted>
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=/srv/dev-disk-by-uuid-6bcdc785-ff84-4338-824f-ac76e5d9695c/docker/immich/upload
TYPESENSE_API_KEY=<redacted>
PUBLIC_LOGIN_PAGE_MESSAGE=<redacted>
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
How do I resolve this?
For me it was the portforwad setting in docker. it stoped working and it was never enabled, very strange.
Similar issue here. I'm using Stacks of Portainer. And here is the error
[Nest] 1 - 05/25/2023, 3:16:45 AM ERROR [TypeOrmModule] Unable to connect to the database. Retrying (1)...
Error: connect ECONNREFUSED 172.19.0.3:5432
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
[Nest] 1 - 05/25/2023, 3:16:48 AM ERROR [TypeOrmModule] Unable to connect to the database. Retrying (2)...
Error: connect ECONNREFUSED 172.19.0.3:5432
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
[Nest] 1 - 05/25/2023, 3:16:51 AM ERROR [TypeOrmModule] Unable to connect to the database. Retrying (3)...
The yml
version: "3.8"
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:release
entrypoint: ["/bin/sh", "./start-server.sh"]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- stack.env
depends_on:
- redis
- database
- typesense
restart: always
immich-microservices:
container_name: immich_microservices
image: ghcr.io/immich-app/immich-server:release
entrypoint: ["/bin/sh", "./start-microservices.sh"]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- stack.env
depends_on:
- redis
- database
- typesense
restart: always
immich-machine-learning:
container_name: immich_machine_learning
image: ghcr.io/immich-app/immich-machine-learning:release
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- model-cache:/cache
env_file:
- stack.env
restart: always
immich-web:
container_name: immich_web
image: ghcr.io/immich-app/immich-web:release
entrypoint: ["/bin/sh", "./entrypoint.sh"]
env_file:
- stack.env
restart: always
typesense:
container_name: immich_typesense
image: typesense/typesense:0.24.0
environment:
- TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
- TYPESENSE_DATA_DIR=/data
logging:
driver: none
volumes:
- tsdata:/data
restart: always
redis:
container_name: immich_redis
image: redis:6.2
restart: always
database:
container_name: immich_postgres
image: postgres:14
env_file:
- stack.env
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
PG_DATA: /var/lib/postgresql/data
volumes:
- pgdata:/var/lib/postgresql/data
restart: always
immich-proxy:
container_name: immich_proxy
image: ghcr.io/immich-app/immich-proxy:release
environment:
# Make sure these values get passed through from the env file
- IMMICH_SERVER_URL
- IMMICH_WEB_URL
ports:
- 2283:8080
depends_on:
- immich-server
restart: always
volumes:
pgdata:
model-cache:
tsdata:
the env vars
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=/mnt/photos/immich
TYPESENSE_API_KEY=some-random-text
PUBLIC_LOGIN_PAGE_MESSAGE=
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
@SunHuawei can you remove the database container and restart the stack?
@alextran1502 , not sure if I get it correctly, however I tried these two ways, and neither of them works.
- Stop immich_postgres, remove it. Stop the stack, and start the stack.
- Step the stack and remove all the images, start the stack again.
@SunHuawei Can you help me run Immich directly from the command line instead of going through Portainer?
@alextran1502 , I don't have docker-compose on my machine. It's OpenWRT, I'm not sure if it can install the docker compose.
@SunHuawei I remember this will help. Remove the stack and remove Postgres volume pg_data. I assume you don't have any data in it, so it will be safe to delete. For some reason, this issue sometimes happens on fresh installation on some systems. Once you started once, it shouldn't happen again
Thanks @alextran1502 , but it is still all the same. I tried
- Remove immich-app_pgdata alone, and restart the stack
- I managed to install
docker compose, and install with it. Same issue. Remove the pgdata, and restart. Same issue. Remove all the volumes, and restart, all the same.
Will that be related to the docker version?
root@OpenWrt:~/immich-app# docker compose version
Docker Compose version v2.18.1
root@OpenWrt:~/immich-app# docker --version
Docker version 20.10.17, build 100c701
@SunHuawei I think as long as you are using the latest docker version you should be good. Ok let's try one last thing, remove everything related to Immich, including the images and pull them again. Finger cross
Still the same. I removed everything, including images and volumes.
@SunHuawei this is strange 🤔
Similar issue here, and for me I had to recreate the db in orfer to get the instance working :) Now I have to figure out how to transfer the old to the new db
I've updated to the latest version a couple of days ago using the commands:
docker compose pull
and
docker compose up -d
in the folder with the docker-compose.yml and the .env file
immich server v1.65.0 running on Debian 11
version: "3.8"
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
command: [ "start.sh", "immich" ]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- .env
depends_on:
- redis
- database
- typesense
restart: always
immich-microservices:
container_name: immich_microservices
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
command: [ "start.sh", "microservices" ]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- .env
depends_on:
- redis
- database
- typesense
restart: always
immich-machine-learning:
container_name: immich_machine_learning
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
volumes:
- model-cache:/cache
env_file:
- .env
restart: always
immich-web:
container_name: immich_web
image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
env_file:
- .env
restart: always
typesense:
container_name: immich_typesense
image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
environment:
- TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
- TYPESENSE_DATA_DIR=/data
logging:
driver: none
volumes:
- tsdata:/data
restart: always
redis:
container_name: immich_redis
image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
restart: always
database:
container_name: immich_postgres
image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
env_file:
- .env
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
PG_DATA: /var/lib/postgresql/data
volumes:
- pgdata:/var/lib/postgresql/data
restart: always
immich-proxy:
container_name: immich_proxy
image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
environment:
# Make sure these values get passed through from the env file
- IMMICH_SERVER_URL
- IMMICH_WEB_URL
ports:
- 2283:8080
depends_on:
- immich-server
restart: always
volumes:
pgdata:
model-cache:
tsdata:
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=#SECRET
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
REDIS_URL=ioredis://eyJzZW50aW5lbHMiOlt7Imhvc3QiOiJyZWRpcy1zZW50aW5lbDEiLCJwb3J0IjoyNjM3OX0seyJob3N0IjoicmVkaXMtc2VudGluZWwyIiwicG9ydCI6MjYzNzl9XSwibmFtZSI6Im15bWFzdGVyIn0=
UPLOAD_LOCATION=/media/samba/immich
TYPESENSE_API_KEY=#SECRET
TYPESENSE_URL=ha://WwogIHsgImhvc3QiOiAidHlwZXNlbnNlLTEuZXhhbXBsZS5uZXQiLCAicG9ydCI6ICI0NDMiLCAicHJvdG9jb2wiOiAiaHR0cHMiIH0sCiAgeyAiaG9zdCI6ICJ0eXBlc2Vuc2UtMi5leGFtcGxlLm5ldCIsICJwb3J0IjogIjQ0MyIsICJwcm90b2NvbCI6ICJodHRwcyIgfSwKICB7ICJob3N0IjogInR5cGVzZW5zZS0zLmV4YW1wbGUubmV0IiwgInBvcnQiOiAiNDQzIiwgInByb3RvY29sIjogImh0dHBzIiB9Cl0=
PUBLIC_LOGIN_PAGE_MESSAGE=
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
I'm getting the error
[Nest] 1 - 07/04/2023, 8:10:28 AM ERROR [TypeOrmModule] Unable to connect to the database. Retrying (1)...
QueryFailedError: could not create unique index "UQ_4ed4f8052685ff5b1e7ca1058ba"
at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async ImportAsset1686584273471.up (/usr/src/app/dist/infra/migrations/1686584273471-ImportAsset.js:10:9)
at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:260:35)
at async DataSource.initialize (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:148:17)
root@debian:~/immich-app# docker compose version
Docker Compose version v2.18.1
root@debian:~/immich-app# docker --version
Docker version 24.0.2, build cb74dfc
@dusnoki please see this thread for the fix https://github.com/immich-app/immich/issues/2931#issuecomment-1605396092
Yes It did work. I actually ran @perfectra1n 's method for multiple rows. Thanks for the help.