immich-app
Photos Web UI page chokes loading many images with same timestamp
The bug
I set up Immich in May 2024, but it has never really loaded more than a few images just when opening the web app, and today I updated to the latest server version and tried to figure out why. It hits 100-300% CPU usage as indicated by docker stats or htop even with 0 running jobs, and the verbose logs below seem to indicate that it's a problem with loading the main /photos HTTP endpoint in a browser. Loading the Photos "timeline" page starts with the most recent, which correctly shows 7 photos with incorrect timestamps from year 4501, but the next most recent 'set' is from May 2024. Maybe Immich thinks there are a ton of images in this set as the log error shows it probably timing out: [Nest] 17 - 10/09/2024, 3:26:44 PM DEBUG [Api:LoggingInterceptor~2rrmvqcs] GET /api/timeline/bucket?isArchived=false&size=MONTH&timeBucket=2024-05-01T00%3A00%3A00.000Z&withPartners=true&withStacked=true 200 37585.19ms ::ffff:172.23.0.1, then [Nest] 17 - 10/09/2024, 3:26:50 PM ERROR [Api:GlobalExceptionFilter~2rrmvqcs] Unknown error: RangeError: Invalid string length. I can't find any images with that timestamp in the folder it's scanning for photos, and after timing out the UI is mostly unresponsive (none of the supposed photos from May 2024 are clickable in the UI, just showing grey boxes (so I can't figure out what Immich thinks these are here either)). As I set up Immich in May 2024 my best guess is it set a default timestamp for tons of pictures and is now choking on loading them all together.
But even if there were tons of photos from this date, I still wouldn't expect it to effectively break Immich - it takes 10+ seconds to open one of the 7 loaded pictures to to try fix the bad metadata dates (which don't save when I edit them), and sometimes fails with a 500. No photos earlier than this load. Same for opening administration setttings. I have 800k photos.
The OS that Immich Server is running on
Ubuntu 22.04
Version of Immich Server
v1.117.0
Version of Immich Mobile App
not used
Platform with the issue
- [X] Server
- [X] Web
- [ ] Mobile
Your docker-compose.yml content
name: immich
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
command: ['start.sh', 'immich']
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- /nas/photo/:/photo:ro
- /etc/localtime:/etc/localtime:ro
env_file:
- .env
ports:
- 2283:3001
depends_on:
- redis
- database
restart: always
immich-microservices:
container_name: immich_microservices
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
# extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
# file: hwaccel.transcoding.yml
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
command: ['start.sh', 'microservices']
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- /nas/photo/:/photo:ro
- /etc/localtime:/etc/localtime:ro
env_file:
- .env
depends_on:
- redis
- database
restart: always
immich-machine-learning:
container_name: immich_machine_learning
# For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
# Example tag: ${IMMICH_VERSION:-release}-cuda
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
# extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
# file: hwaccel.ml.yml
# service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
volumes:
- model-cache:/cache
env_file:
- .env
restart: always
redis:
container_name: immich_redis
image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
restart: always
database:
container_name: immich_postgres
image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
POSTGRES_INITDB_ARGS: '--data-checksums'
volumes:
- ${DB_DATA_LOCATION}:/var/lib/postgresql/data
restart: always
command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
volumes:
model-cache:
Your .env content
# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables
# The location where your uploaded files are stored
UPLOAD_LOCATION=/nas/photo/immich/
# The location where your database files are stored
DB_DATA_LOCATION=./postgres
# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release
DB_PASSWORD=<REDACTED>
# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
Reproduction steps
- Log into Immich Web UI
- Wait for photos to load... and only the ones from year 4501 do.
Relevant log output
[Nest] 17 - 10/09/2024, 3:26:05 PM LOG [Api:EventRepository] Websocket Disconnect: -qhs7ePglVE2q2qOAAAZ
[Nest] 17 - 10/09/2024, 3:26:05 PM DEBUG [Api:LoggingInterceptor~o09ra989] GET /custom.css 200 16.43ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:05 PM DEBUG [Api:LoggingInterceptor~lt2dcova] GET /api/users/me 200 1.19ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:05 PM DEBUG [Api:LoggingInterceptor~lsza7tp5] GET /api/server/features 200 22.36ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:05 PM DEBUG [Api:LoggingInterceptor~hf8o0p16] GET /api/users/me/preferences 200 0.21ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:05 PM DEBUG [Api:LoggingInterceptor~ooh0meeh] GET /api/server/config 200 20.97ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:05 PM DEBUG [Api:LoggingInterceptor~1frary5k] GET /api/server/about 200 143.99ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:05 PM DEBUG [Api:LoggingInterceptor~v3y88z2h] GET /api/server/storage 200 2.46ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:06 PM LOG [Api:EventRepository] Websocket Connect: Yc13jRxuq97wAJ0LAAAb
[Nest] 17 - 10/09/2024, 3:26:06 PM DEBUG [Api:LoggingInterceptor~uyxypdtn] GET /api/assets/memory-lane?day=9&month=10 200 31.03ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:06 PM DEBUG [Api:LoggingInterceptor~w57v5wqc] GET /api/server/storage 200 1.01ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:06 PM DEBUG [Api:LoggingInterceptor~98xc0ypd] GET /api/server/about 200 52.94ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:06 PM DEBUG [Api:LoggingInterceptor~txpta6ut] GET /api/server/version-history 200 1.26ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:06 PM DEBUG [Api:LoggingInterceptor~gddvr1ld] GET /api/server/about 200 46.33ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:06 PM DEBUG [Api:LoggingInterceptor~ofye8rs9] GET /api/timeline/buckets?isArchived=false&size=MONTH&withPartners=true&withStacked=true 200 503.43ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:06 PM DEBUG [Api:LoggingInterceptor~mfne3hmj] GET /api/timeline/bucket?isArchived=false&size=MONTH&timeBucket=4501-01-01T00%3A00%3A00.000Z&withPartners=true&withStacked=true 200 177.83ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:07 PM DEBUG [Api:LoggingInterceptor~2u7hz4ps] GET /api/assets/8451e327-caad-4ecd-8c04-dc1065bab990/thumbnail?size=preview 200 125.04ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:07 PM DEBUG [Api:LoggingInterceptor~5apd2o7w] GET /api/assets/f5024fa4-0ead-43e3-8baf-d3a15ec32051/thumbnail?size=preview 200 293.50ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:07 PM DEBUG [Api:LoggingInterceptor~2nu7fhjf] GET /api/assets/5cb1b066-64c4-4d1b-a31e-97171415fee4/thumbnail?size=preview 200 333.58ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:10 PM DEBUG [Api:LoggingInterceptor~6ph4bhm7] GET /api/server/storage 200 1.20ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:10 PM DEBUG [Api:LoggingInterceptor~2ejvsxjj] GET /api/assets/memory-lane?day=9&month=10 200 133.60ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:10 PM DEBUG [Api:LoggingInterceptor~80gguv5h] GET /api/server/storage 200 27.89ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:10 PM DEBUG [Api:LoggingInterceptor~k7584t0n] GET /api/server/about 200 345.84ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:10 PM DEBUG [Api:LoggingInterceptor~b2e52ttm] GET /api/server/version-history 200 29.47ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:11 PM DEBUG [Api:LoggingInterceptor~5sp91x7m] GET /api/server/about 200 478.15ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:11 PM DEBUG [Api:LoggingInterceptor~ur547xdg] GET /api/timeline/buckets?isArchived=false&size=MONTH&withPartners=true&withStacked=true 200 718.18ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:11 PM DEBUG [Api:LoggingInterceptor~cjyrgdrn] GET /api/albums/statistics 200 64.03ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:11 PM DEBUG [Api:LoggingInterceptor~ih5bcm4r] GET /api/timeline/bucket?isArchived=false&size=MONTH&timeBucket=4501-01-01T00%3A00%3A00.000Z&withPartners=true&withStacked=true 200 217.98ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:44 PM DEBUG [Api:LoggingInterceptor~2rrmvqcs] GET /api/timeline/bucket?isArchived=false&size=MONTH&timeBucket=2024-05-01T00%3A00%3A00.000Z&withPartners=true&withStacked=true 200 37585.19ms ::ffff:172.23.0.1
[Nest] 17 - 10/09/2024, 3:26:50 PM ERROR [Api:GlobalExceptionFilter~2rrmvqcs] Unknown error: RangeError: Invalid string length
### Additional information
_No response_
This is a known issue in that the DB query to fetch timeline assets fetches way too much data. When there are many assets in a bucket, this can lead to the error you're seeing (which I believe means the serialized response is over 512MiB). We intend to slim this down to the bare minimum needed to display the timeline and fetch other asset data as needed.
I see. Would you have any idea why so many photos might have been given a default timestamp of the date I set up Immich? I can't imagine that many of them didn't have any timestamp metadata and would like to identify the culprits to check, but don't know how.
Any other workarounds? I can reset everything immich-db-related and set it up again but am concerned it will repeat the same day metadata problem.
If the assets don't have any date metadata, it'll default to the creation date, i.e. the date of upload. The solution would be to remove the assets, fix the date metadata with exiftool and re-import them. For some test datasets without metadata, I've set random dates for the assets in the DB to spread them out into more buckets. This is an option, but not exactly ideal for a real photo library.
Thanks. I'll look for a way to find photos without date metadata. With 800k photos that's a bit difficult, so I'm curious if there's a way I can quickly look them up in Immich without it crashing in the process. Maybe a sql command I can run on the postgres container?
And for future reference, does immich modify the image file metadata itself when you edit the timestap in the immich UI, or is that only saved to an immich dB (and lost if I view a photo outside immich)?
The relevant column is localDateTime in the assets table.
And for future reference, does immich modify the image file metadata itself when you edit the timestap in the immich UI, or is that only saved to an immich dB (and lost if I view a photo outside immich)?
It's saved in the DB and to an XMP sidecar file. We don't modify the original image as a general principle.
Thanks. When running this query: SELECT * FROM assets WHERE "localDateTime" >= '2024-05-01' AND "localDateTime" < '2024-05-31' LIMIT 10;, all results seem to be thumbnails generated by immich, e.g. filepath is /photo/immich/thumbs/44b614c6-4055-4349-b6cc-140351b7795d/fc/09/fc09158f-40b5-4bff-b476-0b9b55eca6d8.jpeg. So I'm guessing this assets table is for more than just user-imported photos, or maybe I'm not supposed to have immich/ inside the same directory as the rest of the photos (/photo)?
When trying to exclude anything generated by Immich I get no results with this query: SELECT * FROM assets WHERE "localDateTime" >= '2024-05-01' AND "localDateTime" < '2024-05-31' AND "originalPath" NOT LIKE '%immich%' LIMIT 10;. Other partial string queries work so I don't think it's syntax. Searching outside that: WHERE NOT ("localDateTime" >= '2024-05-01' AND "localDateTime" < '2024-05-31') AND "originalPath" LIKE '%immich%' LIMIT 10; (which makes sense since I haven't really tried using Immich since then). But all my other photos show up when excluding filepaths in /immich so I wonder if all these oddly-timestamped files are actually just generated by Immich.
Depending on your external library import path at the time, it might have included the thumbs folder and imported thumbnails as new assets.
I think that's what happened, and maybe is still happening as now I apparently have 1.6 million photos, it might be a never-ending loop of more photos (800k did seem off to me earlier). When setting it up I had added exclusion pattern immich/** but clearly I didn't get that working right so I added **/immich/**, /photo/immich/**, and photo/immich/** too. It would be nice if Immich recognized if its own folder were under an external library path and auto-suggested to add the proper exclusion pattern for future users.
At this point should I just reset the whole thing to get rid of all those thumbnails, or is there a way to keep all the facial recognition buildup etc? Is there an easy way to wipe everything in docker-compose and restart fresh?
If you rescan with the right exclusion filters, it should set the thumbnail assets offline and delete them after the trash period (30 days by default). You can also manually query the DB to set deletedAt to a date and status to 'trashed' and empty the trash in the UI.
Hey is this still an issue? We recently migrated to a new database abstraction layer and rewrote all queries, so most likely this is fixed.
Hi, is anyone currently working on this issue? If not, is there any expectation for when it might be addressed?
It has recently been improved a lot. Are you still experiencing issues @berke0bayraktar ?
@danieldietzler Still present for me. I really do have a lot of files on the exact same date though like tens of thousands. Also I just set up immich so I'm not aware of the previous state.
Edit: I realized im on v1.132.3 I will update and report back
Just updated to the latest v1.134.0 release and there is definitely a lot of improvement. It's still not quite as snappy as part of the timelines that don't have the cluttering problem.
For reporting purposes, specifically when I go to a part of the timeline that has clutter and scroll fast, there is 1-2 second delay where its black screen until the images load. This isn't the case with other parts of the timeline: I can get a continuous scroll.
But its completely usable unlike before.
Yeah for sure. We still don't split up buckets itself if they're too big, so you still get like 40k assets or however many you have in there. We just did a bunch of trimming and optimizing to make it at least usable. Glad it's also much better for you now :)
We've made significant progress on this issue, but there will always be room for improvement. I've decided to track this in #21284, since it is more of a limitation than a bug in the system (the system is working as designed, it's just not designed very well for 10,000 pictures all taken on the same day 😓)
