simulot
Immich-Go Upload Fails for Large Photo Import (120K+ Files)
Hello,
One of my 120,000 photos import keeps erroring out ,a nd not sure how to fix it:
Command: ./immich-go upload from-folder --into-album "Backup" --api-trace --log-file "debug.log" --server=http://192.168.1.90:8080/ --api-key=*********************** "/mnt/photos"
First log lines:
2025-03-23 11:32:17 INF immich-go version:0.25.0, commit:561f56503aa27b5d5870315cfa1a787e3b745dd4, date:2025-03-22T15:56:48Z 2025-03-23 11:32:17 INF Running environment: architecture=amd64 os=linux 2025-03-23 11:32:17 INF Command: immich-go upload from-folder 2025-03-23 11:32:17 INF Flags: 2025-03-23 11:32:17 INF --album-path-joiner= / 2025-03-23 11:32:17 INF --album-picasa=false 2025-03-23 11:32:17 INF --api-key=**********************************girY 2025-03-23 11:32:17 INF --api-trace=true 2025-03-23 11:32:17 INF --ban-file='@eaDir/', '@_thumb/', 'SYNOFILE_THUMB.', 'Lightroom Catalog/', 'thumbnails/', '.DS_Store/', '/._', '.photostructure/' 2025-03-23 11:32:17 INF --client-timeout=5m0s 2025-03-23 11:32:17 INF --date-from-name=true 2025-03-23 11:32:17 INF --date-range=unset 2025-03-23 11:32:17 INF --device-uuid=HomeLab 2025-03-23 11:32:17 INF --dry-run=false 2025-03-23 11:32:17 INF --exclude-extensions= 2025-03-23 11:32:17 INF --folder-as-album=NONE 2025-03-23 11:32:17 INF --folder-as-tags=false 2025-03-23 11:32:17 INF --help=false 2025-03-23 11:32:17 INF --icloud-takeout=false 2025-03-23 11:32:17 INF --ignore-sidecar-files=false 2025-03-23 11:32:17 INF --include-extensions= 2025-03-23 11:32:17 INF --include-type= 2025-03-23 11:32:17 INF --into-album=Backup 2025-03-23 11:32:17 INF --log-file=debug.log 2025-03-23 11:32:17 INF --log-level=INFO 2025-03-23 11:32:17 INF --log-type=text 2025-03-23 11:32:17 INF --manage-burst=NoStack 2025-03-23 11:32:17 INF --manage-epson-fastfoto=false 2025-03-23 11:32:17 INF --manage-heic-jpeg=NoStack 2025-03-23 11:32:17 INF --manage-raw-jpeg=NoStack 2025-03-23 11:32:17 INF --no-ui=false 2025-03-23 11:32:17 INF --on-server-errors=stop 2025-03-23 11:32:17 INF --recursive=true 2025-03-23 11:32:17 INF --server=http://192.168.1.90:8080/ 2025-03-23 11:32:17 INF --session-tag=false 2025-03-23 11:32:17 INF --skip-verify-ssl=false 2025-03-23 11:32:17 INF --tag=[] 2025-03-23 11:32:17 INF --time-zone= 2025-03-23 11:32:17 INF Arguments: 2025-03-23 11:32:17 INF "/mnt/photos" 2025-03-23 11:32:17 INF Connection to the server http://192.168.1.90:8080 2025-03-23 11:32:17 INF Server status: OK 2025-03-23 11:32:18 INF Server information: version=v1.129.0 2025-03-23 11:32:18 INF Connected, user: XXXX, ID: XXXX 2025-03-23 11:32:18 INF Check the API-TRACE file: debug.trace.log
Last lines:
2025-03-23 13:39:48 INF server has same asset file=photos:backup/Recovered data 03-04 11_09_15/Deep Scan result/Special Lost Files(Lab)/Camera/FUJIFILM/FinePix265021927.JPG reason=An asset with the same name:"FinePix265021925.JPG", date:"2002-06-24 00:39:32" and size:308.8 KB exists on the server. No need to upload
2025-03-23 13:40:55 INF 2025-03-23 13:40:55 INF Input analysis: 2025-03-23 13:40:55 INF --------------- 2025-03-23 13:40:55 INF scanned image file : 63547 2025-03-23 13:40:55 INF scanned video file : 354 2025-03-23 13:40:55 INF scanned sidecar file : 3 2025-03-23 13:40:55 INF discarded file : 1 2025-03-23 13:40:55 INF unsupported file : 0 2025-03-23 13:40:55 INF file duplicated in the input : 9 2025-03-23 13:40:55 INF associated metadata file : 0 2025-03-23 13:40:55 INF missing associated metadata file : 0 2025-03-23 13:40:55 INF 2025-03-23 13:40:55 INF Uploading: 2025-03-23 13:40:55 INF ---------- 2025-03-23 13:40:55 INF uploaded : 0 2025-03-23 13:40:55 INF upload error : 0 2025-03-23 13:40:55 INF file not selected : 0 2025-03-23 13:40:55 INF server's asset upgraded with the input : 0 2025-03-23 13:40:55 INF server has same asset : 55291 2025-03-23 13:40:55 INF server has a better asset : 0 2025-03-23 13:40:55 INF 2025-03-23 13:40:55 ERR Some errors have occurred. Look at the log file for details
Here's the API Trace log:
2025-03-23T13:39:21-04:00 RESPONSE 64 createStack POST http://192.168.1.90:8080/api/stacks Header: Keep-Alive : timeout=5 X-Powered-By : Express X-Immich-Cid : 9rq5ly0p Content-Type : application/json; charset=utf-8 Content-Length : 2561 Etag : "a01-leg8Fe0TyFIkUteLYI86TGfIE+8" Date : Sun, 23 Mar 2025 17:39:21 GMT Connection : keep-alive Status: 201 Created -- response body start -- {"id":"xxx","primaryAssetId":"a651521e-c09d-4356-bb9d-2d272b03227d","assets":[{"id":"a651521e-c09d-4356-bb9d-2d272b03227d","deviceAssetId":"WpdThumbnail.png-8831","ownerId":"535cff05-2f35-4543-99ee-1ef9c46f33af","deviceId":"xxx","libraryId":null,"type":"IMAGE","originalPath":"/photos/library/shared/2012/2012-03-07/WpdThumbnail.png","originalFileName":"WpdThumbnail.png","originalMimeType":"image/png","thumbhash":"8+cJFwLJt5h5h3dvhVaH12iHhArnyXUP","fileCreatedAt":"2012-03-07T09:21:00+00:00","fileModifiedAt":"2012-03-07T09:21:00+00:00","localDateTime":"2012-03-07T09:21:00+00:00","updatedAt":"2025-03-23T17:39:21.736117+00:00","isFavorite":false,"isArchived":false,"isTrashed":false,"duration":"0:00:00.00000","exifInfo":{"make":null,"model":null,"exifImageWidth":96,"exifImageHeight":96,"fileSizeInByte":8831,"orientation":null,"dateTimeOriginal":"2012-03-07T09:21:00+00:00","modifyDate":"2012-03-07T09:21:00+00:00","timeZone":null,"lensModel":null,"fNumber":null,"focalLength":null,"iso":null,"exposureTime":null,"latitude":null,"longitude":null,"city":null,"state":null,"country":null,"description":"","projectionType":null,"rating":null},"livePhotoVideoId":null,"people":[],"checksum":"JzSca0aWkxzSkfFi81fRlTSs52M=","isOffline":false,"hasMetadata":true,"duplicateId":null,"resized":true},{"id":"7644c76b-43cf-485f-a57e-31ced3859f21","deviceAssetId":"WpdThumbnail.tiff-56860","ownerId":"xxx","deviceId":"xxx","libraryId":null,"type":"IMAGE","originalPath":"/photos/library/shared/2007/2007-07-25/WpdThumbnail.tiff","originalFileName":"WpdThumbnail.tiff","originalMimeType":"image/tiff","thumbhash":"8+cJFwLJtph5h3dvhVaH12iHhArnyXUP","fileCreatedAt":"2007-07-25T15:22:50+00:00","fileModifiedAt":"2012-03-07T09:21:00+00:00","localDateTime":"2007-07-25T11:22:50+00:00","updatedAt":"2025-03-23T17:39:21.73538+00:00","isFavorite":false,"isArchived":false,"isTrashed":false,"duration":"0:00:00.00000","exifInfo":{"make":null,"model":null,"exifImageWidth":96,"exifImageHeight":96,"fileSizeInByte":56860,"orientation":"1","dateTimeOriginal":"2007-07-25T15:22:50+00:00","modifyDate":"2007-07-25T15:22:50+00:00","timeZone":"UTC-4","lensModel":null,"fNumber":null,"focalLength":null,"iso":null,"exposureTime":null,"latitude":null,"longitude":null,"city":null,"state":null,"country":null,"description":"","projectionType":null,"rating":null},"livePhotoVideoId":null,"people":[],"checksum":"3B9jqNF6qZzpQfq9d4/ZqE2MXPg=","isOffline":false,"hasMetadata":true,"duplicateId":null,"resized":true}]}
I've looked through the log but I don't see anything that could be wrong, I tried yesterday and it gave me a 500 internal error. I have also ran one with --dry-run and it completed just fine.
Thank you
The dry-run doesn't write anything on the server. As no action is posted to the server, errors are less prone to come.
This log part doesn't show any errors. Could you search for ERR in the immich-go log file?
The summary reports shows 63547 photos + 354 videos, but only 55291 processed files (already present on the server), were are the others? I'd need the full log for investigation.
The api trace part shows a success. Was it the last call in the log? The call was for stacking JPG and a TIFF, despite the --manage-raw-jpeg=NoStack option, and the time difference between the two files.
The summary reports shows 63547 photos + 354 videos, but only 55291 processed files (already present on the server), were are the others? I'd need the full log for investigation.
After getting an error the program asks me to quit. This was the all the files it scanned before asking me to exit.
The api trace part shows a success. Was it the last call in the log?
Yes that was the last call in the log despite it showing success. That's why I am having trouble troubleshooting the issue since there doesn't appear to be anything wrong with it.
Btw that log was not from the run I did use dry-run so all actions were posted. I'm gonna try to delete all the files and try again. I will post the new log to see if it improved
Here is the debug.log I had to stop it in the middle but restarted it back up an hour later. I have looked through the file and the only Error i see is "ERR ReadDir not a directory".
Here's the API trace.log. I don't see anything wrong with it but I could've missed something.
Please let me know what the next step should be.
ReadDir $RECYCLE.BIN/S-1-5-21-1796673143-239304482-1539857752-4545/$RTDX8NL.artist_states....
I think you are mounting a windows share, and some windows' hidden directory at share root are not accessible... Could you move your photo into a sub directory and upload from /mnt/photos/subdir ?
ERR stat backup/Recovered data 03-04 11_09_15/Deep Scan result/Special Lost Files(Lab)/Camera/APPLE/iPhone 41970.JPG: input/output error
I can't tell more than the error message: IO error. Is you disk reliable?
You have to know that uploading from a remote folder can be very slow and error prone depending on you configuration. The best option is to run immich-go on the machine that owns the takeout while accessing to the remote immich. The reason is that SMB share is inefficient and the http connection to immich is designed for remote access.
So I have all my photos on an external SSD and they are scattered throughout the SSD into various directories, so putting all the photos into the same sub directory is not really an option.
The disks is reliable after running some tests.
The best option is to run immich-go on the machine that owns the takeout while accessing to the remote immich. The reason is that SMB share is inefficient and the http connection to immich is designed for remote access.
What do you mean by this? How would I set this up?
I hit something similar - I think as the uploads are happening, Immich starts transcoding/generating thumbnails/etc which bogs down the server. My solution was to allow all the jobs to begin and pause them immediately so they stay paused. Then resume them after upload is complete. Seems to be working well so far
