simulot
AssetReplace call PUT http://localhost:2283/api/assets/a963f37e-5db2-4930-9517-36e41fa15c01/original generates an error 500
is there anything one can do to help fix this issue?
I've run into the same problem and unfortunately it entirely stops the process of uploading anything. This is a huge pain for me 😄
│2025-04-28 15:14:22 ERR upload error file=takeout-20250423T134039Z-006:Takeout/Google Photos/Photos from 2013/20130528_125235.jpg error=AssetReplace, PUT, http://127.0.0.1:2283/api/assets/a7613b06-7ba8-412d-9b67-37fd3becf93e/original, 500 Internal Server Error │ │Failed to replace asset │ │ │ │2025-04-28 15:14:22 INF Stacked file=takeout-20250423T134039Z-006:Takeout/Google Photos/Photos from 2013/20130528_125235-edited.jpg │ │2025-04-28 15:14:22 INF Stacked file=takeout-20250423T134039Z-006:Takeout/Google Photos/Photos from 2013/20130528_125235.jpg │ │2025-04-28 15:14:22 ERR AssetReplace, PUT, http://127.0.0.1:2283/api/assets/a7613b06-7ba8-412d-9b67-37fd3becf93e/original, 500 Internal Server Error │ │Failed to replace asset
The backend log doesnt show something unusual. The missing input file happens regularly unfortunately. But I didn't think it's a big deal because you could rerun the job once everything is imported
immich_server | [Nest] 7 - 04/28/2025, 3:14:21 PM ERROR [Microservices:{"id":"a7613b06-7ba8-412d-9b67-37fd3becf93e","source":"upload"}] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: Input file is missing: upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/17/ae/17aea8c7-697c-4ac7-b35e-c3a8982c27ef.jpg immich_server | Error: Input file is missing: upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/17/ae/17aea8c7-697c-4ac7-b35e-c3a8982c27ef.jpg immich_server | at Sharp.toBuffer (/usr/src/app/node_modules/sharp/lib/output.js:163:17) immich_server | at MediaRepository.decodeImage (/usr/src/app/dist/repositories/media.repository.js:97:68) immich_server | at MediaService.generateImageThumbnails (/usr/src/app/dist/services/media.service.js:192:59) immich_server | at process.processTicksAndRejections (node:internal/process/task_queues:105:5) immich_server | at async MediaService.handleGenerateThumbnails (/usr/src/app/dist/services/media.service.js:114:25) immich_server | at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:156:28) immich_server | at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:126:13) immich_server | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) immich_server | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
When I try to GET the URL from the failed PUT I get a http500 as well
GET http://127.0.0.1:2283/api/assets/a7613b06-7ba8-412d-9b67-37fd3becf93e/original 500 (Internal Server Error)
I'll rerun the import with --log-level=DEBUG and --api-trace and post what comes up
debug log
2025-04-28 16:03:59 INF uploaded file=takeout-20250423T134039Z-006:Takeout/Google Photos/Photos from 2013/20130608_192905-edited.jpg 2025-04-28 16:03:59 DBG file.FileName=takeout-20250423T134039Z-006:Takeout/Google Photos/Photos from 2013/20130608_192905-edited.jpg file.FileDate=2025-04-23 14:18:44 file.Description= file.Title=20130608_192905.jpg file.FileSize=553647 file.ID=ffa97f24-3c75-4f75-a7de-e710f5746f6f file.CaptureDate=2013-06-08 19:29:05 file.Trashed=false file.Archived=false file.FromPartner=false file.Favorite=false file.Stars=0 file.Latitude=0.xxxxx file.Longitude=0.xxxxx 2025-04-28 16:03:59 INF Tagged file=takeout-20250423T134039Z-006:Takeout/Google Photos/Photos from 2013/20130608_192905-edited.jpg tag=takeout-20250423T134039Z 2025-04-28 16:03:59 DBG CacheReader: remove temporary file name=C:\Users\lsc\AppData\Local\immich-go\temp\immich-go_4089220007 2025-04-28 16:03:59 ERR upload error file=takeout-20250423T134039Z-006:Takeout/Google Photos/Photos from 2013/20130608_192905.jpg error=AssetReplace, PUT, http://127.0.0.1:2283/api/assets/ffa97f24-3c75-4f75-a7de-e710f5746f6f/original, 500 Internal Server Error Failed to replace asset 2025-04-28 16:03:59 DBG replaced by ID=ffa97f24-3c75-4f75-a7de-e710f5746f6f file.FileName=takeout-20250423T134039Z-006:Takeout/Google Photos/Photos from 2013/20130608_192905.jpg file.FileDate=2025-04-23 14:17:26 file.Description= file.Title=20130608_192905.jpg file.FileSize=2122274 file.ID= file.CaptureDate=2013-06-08 19:29:05 file.Trashed=false file.Archived=false file.FromPartner=false file.Favorite=false file.Stars=0 file.Latitude=0.xxxxx file.Longitude=0.xxxxx 2025-04-28 16:03:59 DBG CacheReader: remove temporary file name=C:\Users\lsc\AppData\Local\immich-go\temp\immich-go_3337733408 2025-04-28 16:03:59 INF Stacked file=takeout-20250423T134039Z-006:Takeout/Google Photos/Photos from 2013/20130608_192905-edited.jpg 2025-04-28 16:03:59 INF Stacked file=takeout-20250423T134039Z-006:Takeout/Google Photos/Photos from 2013/20130608_192905.jpg 2025-04-28 16:03:59 ERR AssetReplace, PUT, http://127.0.0.1:2283/api/assets/ffa97f24-3c75-4f75-a7de-e710f5746f6f/original, 500 Internal Server Error Failed to replace asset
Trace log
2025-04-28T16:03:59+02:00 QUERY 2485 AssetUpload POST http://127.0.0.1:2283/api/assets Accept [application/json] X-Immich-Checksum [QX7FG+h90Kh7Bl5awpzLuNLPutU=] Content-Type [multipart/form-data; boundary=9006460a87025c51e52828fab3413bafb149189df4bed3b6c16f06fd9040] X-Api-Key redacted Post values assetType: image deviceAssetId: 20130608_192905.jpg-553647 deviceId: DESKTOP-A3BGJBQ duration: 00:00:00.000000 fileCreatedAt: 2013-06-08T19:29:05Z fileExtension: .jpg fileModifiedAt: 2025-04-28T14:03:59Z isArchived: false isFavorite: false isReadOnly: false -- binary body not dumped -- 2025-04-28T16:03:59+02:00 RESPONSE 2485 AssetUpload POST http://127.0.0.1:2283/api/assets Header: Keep-Alive : timeout=5 X-Powered-By : Express X-Immich-Cid : x4mpep46 Content-Type : application/json; charset=utf-8 Content-Length : 64 Etag : "40-4Y9HRtSZ6fo44d7Im3S3deInbNQ" Date : Mon, 28 Apr 2025 14:03:57 GMT Connection : keep-alive Status: 201 Created -- response body start -- {"id":"ffa97f24-3c75-4f75-a7de-e710f5746f6f","status":"created"} -- response body end -- 2025-04-28T16:03:59+02:00 QUERY 2486 updateAsset PUT http://127.0.0.1:2283/api/assets/ffa97f24-3c75-4f75-a7de-e710f5746f6f Content-Type [application/json] X-Api-Key redacted -- body start -- {"dateTimeOriginal":"2013-06-08T19:29:05+02:00"} -- body end -- 2025-04-28T16:03:59+02:00 RESPONSE 2486 updateAsset PUT http://127.0.0.1:2283/api/assets/ffa97f24-3c75-4f75-a7de-e710f5746f6f Header: Etag : "4c7-XeD2+LBoo+i67LZXo1CG9YW68XM" Date : Mon, 28 Apr 2025 14:03:57 GMT Connection : keep-alive Keep-Alive : timeout=5 X-Powered-By : Express X-Immich-Cid : n3ihdrp9 Content-Type : application/json; charset=utf-8 Content-Length : 1223 Status: 200 OK -- response body start -- {"id":"ffa97f24-3c75-4f75-a7de-e710f5746f6f","deviceAssetId":"20130608_192905.jpg-553647","ownerId":"1fe57f24-7c2c-46f1-a2cd-9581d09721a1","deviceId":"DESKTOP-A3BGJBQ","libraryId":null,"type":"IMAGE","originalPath":"upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/22/1b/221bc08c-ff2f-4f88-a2f0-7dcdfed75b40.jpg","originalFileName":"20130608_192905.jpg","originalMimeType":"image/jpeg","thumbhash":null,"fileCreatedAt":"2013-06-08T19:29:05.000Z","fileModifiedAt":"2025-04-28T14:03:59.000Z","localDateTime":"2013-06-08T19:29:05.000Z","updatedAt":"2025-04-28T14:03:57.870Z","isFavorite":false,"isArchived":false,"isTrashed":false,"duration":"00:00:00.000000","exifInfo":{"make":null,"model":null,"exifImageWidth":null,"exifImageHeight":null,"fileSizeInByte":553647,"orientation":null,"dateTimeOriginal":"2013-06-08T17:29:05+00:00","modifyDate":null,"timeZone":null,"lensModel":null,"fNumber":null,"focalLength":null,"iso":null,"exposureTime":null,"latitude":null,"longitude":null,"city":null,"state":null,"country":null,"description":"","projectionType":null,"rating":null},"livePhotoVideoId":null,"people":[],"checksum":"QX7FG+h90Kh7Bl5awpzLuNLPutU=","isOffline":false,"hasMetadata":true,"duplicateId":null,"resized":true} -- response body end -- 2025-04-28T16:03:59+02:00 QUERY 2487 AssetReplace PUT http://127.0.0.1:2283/api/assets/ffa97f24-3c75-4f75-a7de-e710f5746f6f/original Accept [application/json] X-Immich-Checksum [f6IBRLDXXzy9jrhq75w7ku0zfgw=] Content-Type [multipart/form-data; boundary=81dadc6ea0759d7d1c6548eda98ef31dcbad1c90fddb2bf66483d074c21e] X-Api-Key redacted Post values assetType: image deviceAssetId: 20130608_192905.jpg-2122274 deviceId: DESKTOP-A3BGJBQ duration: 00:00:00.000000 fileCreatedAt: 2013-06-08T19:29:05Z fileExtension: .jpg fileModifiedAt: 2025-04-28T14:03:59Z isArchived: false isFavorite: false isReadOnly: false -- binary body not dumped -- 2025-04-28T16:03:59+02:00 RESPONSE 2487 AssetReplace PUT http://127.0.0.1:2283/api/assets/ffa97f24-3c75-4f75-a7de-e710f5746f6f/original Status: 500 Internal Server Error -- response body -- -- response body end --
Backend log
immich_server | [Nest] 7 - 04/28/2025, 4:03:56 PM ERROR [Microservices:{"id":"176da240-b070-4dd5-bd71-92b9455cf7da","source":"sidecar-write"}] Unable to run job handler (metadataExtraction/metadata-extraction): Error: ENOENT: no such file or directory, stat 'upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/67/de/67de16ac-7a22-42d8-9dc8-2bdbe6811960.jpg' immich_server | Error: ENOENT: no such file or directory, stat 'upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/67/de/67de16ac-7a22-42d8-9dc8-2bdbe6811960.jpg' immich_server | at async Object.stat (node:internal/fs/promises:1036:18) immich_server | at async Promise.all (index 1) immich_server | at async MetadataService.handleMetadataExtraction (/usr/src/app/dist/services/metadata.service.js:129:35) immich_server | at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:156:28) immich_server | at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:126:13) immich_server | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) immich_server | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) immich_server | [Nest] 7 - 04/28/2025, 4:03:56 PM WARN [Microservices:Error: File not found - /usr/src/app/upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/67/de/67de16ac-7a22-42d8-9dc8-2bdbe6811960.jpg immich_server | at ReadTask._ExifToolTask_parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:76:71) immich_server | at ReadTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:47:167) immich_server | at ReadTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)] Error reading exif data (upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/67/de/67de16ac-7a22-42d8-9dc8-2bdbe6811960.jpg): Error: File not found - /usr/src/app/upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/67/de/67de16ac-7a22-42d8-9dc8-2bdbe6811960.jpg immich_server | [Nest] 7 - 04/28/2025, 4:03:56 PM LOG [Microservices:PersonService] Detected 1 new faces in asset f4d5acf7-f4a7-429e-a914-74cc2bb6ac08 immich_server | [Nest] 7 - 04/28/2025, 4:03:56 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 402e34a2-a498-4842-8394-54c72fb687b5 immich_server | [Nest] 7 - 04/28/2025, 4:03:56 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 84c09e41-7730-4e08-9482-411f580fff3a immich_server | [Nest] 7 - 04/28/2025, 4:03:56 PM WARN [Microservices:Error: File not found - /usr/src/app/upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/ef/c0/efc0b5de-b162-42cf-980c-e9296839ece7.jpg immich_server | at ReadTask._ExifToolTask_parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:76:71) immich_server | at ReadTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:47:167) immich_server | at ReadTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)] Error reading exif data (upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/ef/c0/efc0b5de-b162-42cf-980c-e9296839ece7.jpg): Error: File not found - /usr/src/app/upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/ef/c0/efc0b5de-b162-42cf-980c-e9296839ece7.jpg immich_server | [Nest] 7 - 04/28/2025, 4:03:56 PM LOG [Microservices:PersonService] Detected 1 new faces in asset c5114e18-dced-4ebb-a4e6-cf36c0cf6075 immich_server | [Nest] 7 - 04/28/2025, 4:03:57 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 8eab2dd9-a624-4c12-8893-e29fea2c2183 immich_server | [Nest] 7 - 04/28/2025, 4:03:57 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 82328f28-f003-4c00-8a47-b42dcbb7dc8f immich_server | [Nest] 7 - 04/28/2025, 4:03:57 PM WARN [Microservices:Error: File not found - /usr/src/app/upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/3a/19/3a194893-62a4-466e-ba6a-a9bc0c8f6a07.jpg immich_server | at ReadTask._ExifToolTask_parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:76:71) immich_server | at ReadTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:47:167) immich_server | at ReadTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)] Error reading exif data (upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/3a/19/3a194893-62a4-466e-ba6a-a9bc0c8f6a07.jpg): Error: File not found - /usr/src/app/upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/3a/19/3a194893-62a4-466e-ba6a-a9bc0c8f6a07.jpg immich_server | [Nest] 7 - 04/28/2025, 4:03:58 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 9e38f4cf-97de-4e48-a05a-a8d522a4cd1e immich_server | [Nest] 19 - 04/28/2025, 4:03:58 PM ERROR [Api:AssetMediaService~fqpo2osj] Error uploading file Error: ENOENT: no such file or directory, stat 'upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/22/1b/221bc08c-ff2f-4f88-a2f0-7dcdfed75b40.jpg' immich_server | Error: ENOENT: no such file or directory, stat 'upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/22/1b/221bc08c-ff2f-4f88-a2f0-7dcdfed75b40.jpg' immich_server | at async Object.stat (node:internal/fs/promises:1036:18) immich_server | at async AssetMediaService.createCopy (/usr/src/app/dist/services/asset-media.service.js:262:26) immich_server | at async AssetMediaService.replaceAsset (/usr/src/app/dist/services/asset-media.service.js:114:33) immich_server | at async AssetMediaController.replaceAsset (/usr/src/app/dist/controllers/asset-media.controller.js:51:29) immich_server | [Nest] 19 - 04/28/2025, 4:03:58 PM ERROR [Api:ErrorInterceptor~fqpo2osj] Unknown error: Error: ENOENT: no such file or directory, stat 'upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/22/1b/221bc08c-ff2f-4f88-a2f0-7dcdfed75b40.jpg' immich_server | Error: ENOENT: no such file or directory, stat 'upload/upload/1fe57f24-7c2c-46f1-a2cd-9581d09721a1/22/1b/221bc08c-ff2f-4f88-a2f0-7dcdfed75b40.jpg' immich_server | at async Object.stat (node:internal/fs/promises:1036:18) immich_server | at async AssetMediaService.createCopy (/usr/src/app/dist/services/asset-media.service.js:262:26) immich_server | at async AssetMediaService.replaceAsset (/usr/src/app/dist/services/asset-media.service.js:114:33) immich_server | at async AssetMediaController.replaceAsset (/usr/src/app/dist/controllers/asset-media.controller.js:51:29) immich_server | [Nest] 7 - 04/28/2025, 4:03:58 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 176da240-b070-4dd5-bd71-92b9455cf7da immich_server | [Nest] 7 - 04/28/2025, 4:03:58 PM LOG [Microservices:PersonService] Detected 3 new faces in asset 381f8266-439c-4699-949e-0713ee92b574 immich_server | [Nest] 7 - 04/28/2025, 4:03:58 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 67e30f7b-926c-459f-b64a-aa43d6bedba6 immich_server | [Nest] 7 - 04/28/2025, 4:03:58 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 2bebb552-f5ee-485c-a41a-1603fac846fe immich_server | [Nest] 7 - 04/28/2025, 4:03:58 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 6deec474-4bb5-46e6-b100-1896f217612e immich_server | [Nest] 7 - 04/28/2025, 4:03:58 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 6bde4a2d-0462-4a9f-ac02-a465943af4e8 immich_server | [Nest] 7 - 04/28/2025, 4:03:58 PM LOG [Microservices:PersonService] Detected 1 new faces in asset c3598805-e138-4baa-bd4b-5a98ace6ae41 immich_server | [Nest] 7 - 04/28/2025, 4:03:59 PM LOG [Microservices:PersonService] Detected 1 new faces in asset ae3e3711-4a2a-429d-b315-b5baac581802 immich_server | [Nest] 7 - 04/28/2025, 4:03:59 PM LOG [Microservices:PersonService] Detected 4 new faces in asset 4d46bcde-d817-41fb-a370-eac620686159 immich_server | [Nest] 7 - 04/28/2025, 4:03:59 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 923599c4-7dc3-4d29-98be-fd76565b26ad immich_server | [Nest] 7 - 04/28/2025, 4:03:59 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 5c170543-534f-4fbb-a5b2-306b9aaf4643 immich_server | [Nest] 7 - 04/28/2025, 4:03:59 PM LOG [Microservices:PersonService] Detected 1 new faces in asset 4d074a9c-c619-4a93-804e-5926a7d9c905 immich_server | [Nest] 7 - 04/28/2025, 4:03:59 PM LOG [Microservices:PersonService] Detected 1 new faces in asset cb74a995-204d-4a48-ac6a-ad17d1799096 immich_server | [Nest] 7 - 04/28/2025, 4:04:00 PM LOG [Microservices:PersonService] Creating new person for face 5517ebfc-1e31-47f8-a287-b6366e98eaeb immich_server | [Nest] 7 - 04/28/2025, 4:04:01 PM LOG [Microservices:PersonService] Creating new person for face dfe2dd8a-ea6a-47ea-97cc-ffa0ef26fbe4 immich_redis | 1:M 28 Apr 2025 15:00:15.996 * 100 changes in 300 seconds. Saving...
You must change the server's log level to DEBUG as well to see what is going wrong on the server side
Which version of immich-go and immich?
I just wanted to pitch in and say I'm having this issue as well. Something must have changed in the past month or two, because I did a google takeout upload using immich-go (same exact version) before and never had any problems. I don't imagine that information is too useful though.
Either way, I'm not sure where to go for help on this: immich-go or immich.
I do have debug logs but I can't figure out how to pipe docker compose logs -f immich > immich.log without ASCII escape sequences. If you have a solution for that let me know -- I haven't had much luck googling for one.
I'm happy to provide my debug logs after that.
There are lot of moving pieces. The 1st is your immich server content, then the software version, and the takeout format.
You can share your logs privately on my discord account @simulot.
immich debug log activated
I think comment formatting would ruin the text so I just post a screenshot..
I chatted with the immich devs last night and they asked me to file this issue: https://github.com/immich-app/immich/issues/17954
I think the issue is server side. I will still message you the debug logs.
This make some sense. Immich-go uploads a file, and then updates its metadata. In the meantime, an Immich's background process takes the file and moves it.
Would it be possible to include some workaround in immich-go? Like.. repeat the request a couple of times, if it has failed? idk if that makes sense. But I really hope to get my upload done. Wouldn't mind to hack something into the code and recompile it 😊
No, immich-go can't control when the back ground job starts, nor if the jobs are done in the same order as the photos are uploaded.
I think we need some help from Immich team. To be sure that immich-go won't interfere with any background job, immich-go should provide all information in one call... Then immich can process the information at its will.
I understand. But wouldn't it be possible to just skip the file and leave it as not-uploaded so I can upload the other files?
Have you tried the option
--on-server-errors=continue?
This seems to work for me. 600Gb+ Google Takeout upload.
