immich
immich copied to clipboard
Published
20 hours ago •
immich-app
immich-app
[BUG]: Corrupted Reverse Geocoding CSV File
https://github.com/tomayac/local-reverse-geocoder/issues/63
Possible solutions:
- Fix upstream
- Detect the issue, delete the csv file/directory, and try again
Hello, just wanted to share that this keeps failing for me. Even after recreating the microservices container I get this error shortly after starting it:
[Nest] 1 - 04/06/2023, 7:10:41 PM LOG [MediaService] Start encoding video 417b1bc3-471c-4533-869b-98c00d9f35bb -crf 23,-preset ultrafast,-vcodec h264,-acodec aac,-movflags faststart,-vf scale=-2:720
[Nest] 1 - 04/06/2023, 7:10:41 PM LOG [MediaService] Start encoding video a8f851a9-e466-498a-bb44-5f597891b029 -crf 23,-preset ultrafast,-vcodec h264,-acodec aac,-movflags faststart,-vf scale=-2:720
/usr/src/app/node_modules/local-reverse-geocoder/index.js:746
throw err;
^
Error downloading GeoNames admin 2 codes data: Error: read ECONNRESET
(Use `node --trace-uncaught ...` to show where the exception was thrown)
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [NestFactory] Starting Nest application...
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +213ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +3ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] ConfigModule dependencies initialized +23ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] BullModule dependencies initialized +2ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +543ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
[Nest] 1 - 04/06/2023, 7:12:29 PM WARN [MetadataExtractionProcessor] Reverse geocoding is enabled
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] InfraModule dependencies initialized +18ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] DomainModule dependencies initialized +1ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [NestApplication] Nest application successfully started +49ms
[Nest] 1 - 04/06/2023, 7:12:29 PM LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.53.0 - Listening on port: 3002
/usr/src/app/node_modules/local-reverse-geocoder/index.js:746
throw err;
^
CsvError: Invalid Record Length: expect 19, got 14 on line 10326
at Object.__onRecord (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:940:11)
at Object.parse (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:903:36)
at Parser._flush (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:1336:26)
at Parser.final [as _final] (node:internal/streams/transform:112:25)
at callFinal (node:internal/streams/writable:694:27)
at prefinish (node:internal/streams/writable:723:7)
at finishMaybe (node:internal/streams/writable:733:5)
at afterWrite (node:internal/streams/writable:504:3)
at onwrite (node:internal/streams/writable:477:7)
at Parser.Transform._read (node:internal/streams/transform:245:5) {
code: 'CSV_RECORD_INCONSISTENT_FIELDS_LENGTH',
bytes: 1589248,
comment_lines: 0,
empty_lines: 0,
invalid_field_length: 0,
lines: 10326,
records: 10325,
columns: false,
error: undefined,
header: false,
index: 14,
raw: undefined,
column: 14,
quoting: false,
record: [
'2801970', 'Biesme',
'Biesme', 'Mettet (Biesme)',
'50.34015', '4.61048',
'P', 'PPL',
'BE', '',
'WAL', 'WNA',
'92', '92'
]
}
This has started happening after performing a batch upload with the CLI of all my media from 2021. I am now left in a state where the media is uploaded (it's in the upload/{USER-UID} directory) but is unable to be processed.
I tried getting the cities500.zip from inside the container and it seems to work fine:
/usr/src/app # wget https://download.geonames.org/export/dump/cities500.zip
Connecting to download.geonames.org (5.9.152.54:443)
saving to 'cities500.zip'
cities500.zip 100% |**********************************************************************************************************************************************| 10.0M 0:00:00 ETA
'cities500.zip' saved
/usr/src/app #
I don't have any geocoding-related env variables, so I'm assuming it's using as default REVERSE_GEOCODING_PRECISION=3.
I've recreated the container endless times now but the result is always the same.
EDIT: slighly different error I just got:
/usr/src/app/node_modules/local-reverse-geocoder/index.js:746
throw err;
^
Error downloading GeoNames cities1000 data: Error: aborted
(Use `node --trace-uncaught ...` to show where the exception was thrown)
Hello, just wanted to share that this keeps failing for me. Even after recreating the microservices container I get this error shortly after starting it:
same for me! I also did a CLI batch upload of various years.
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +211ms
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [InstanceLoader] BullModule dependencies initialized +2ms
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +3ms
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [InstanceLoader] ConfigModule dependencies initialized +26ms
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [InstanceLoader] BullModule dependencies initialized +2ms
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +422ms
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +2ms
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
[Nest] 1 - 05/03/2023, 4:44:23 PM WARN [MetadataExtractionProcessor] Reverse geocoding is enabled
[Nest] 1 - 05/03/2023, 4:44:23 PM LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
/usr/src/app/node_modules/local-reverse-geocoder/index.js:746
throw err;
^
CsvError: Invalid Record Length: expect 19, got 15 on line 179072
at Object.__onRecord (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:940:11)
at Object.parse (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:903:36)
at Parser._flush (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:1336:26)
at Parser.final [as _final] (node:internal/streams/transform:112:25)
at callFinal (node:internal/streams/writable:694:27)
at prefinish (node:internal/streams/writable:723:7)
at finishMaybe (node:internal/streams/writable:733:5)
at afterWrite (node:internal/streams/writable:504:3)
at onwrite (node:internal/streams/writable:477:7)
at Parser.Transform._read (node:internal/streams/transform:245:5) {
code: 'CSV_RECORD_INCONSISTENT_FIELDS_LENGTH',
bytes: 28737536,
comment_lines: 0,
empty_lines: 0,
invalid_field_length: 0,
lines: 179072,
records: 179071,
columns: false,
error: undefined,
header: false,
index: 15,
raw: undefined,
column: 15,
quoting: false,
record: [
'4295878',
'Inez',
'Inez',
'Ajnez,Eden,Inez,anyz,aynz kntaky,yi nei ci,Айнез,Ајнез,إنيز,اینز، کنتاکی,伊内兹',
'37.86648',
'-82.53876',
'P',
'PPLA2',
'US',
'',
'KY',
'159',
'',
'',
'6'
]
}```
this is the last line in the cities500 textfile.
If I change it, the error vanishes
```4295878 Inez Inez Ajnez,Eden,Inez,anyz,aynz kntaky,yi nei ci,Айнез,Ајнез,إنيز,اینز، کنتاکی,伊内兹 37.86648 -82.53876 P PPLA2 US KY 159 6 180 184 America/Kentucky/Louisville 2006-01-15
where as "180 184 America/Kentucky/Louisville 2006-01-15" is just copied from the second last line.
After that the container still crashes... with this error:
Log in reverse order:
2023-05-03T14:59:27.940504004Z stderr [31m[Nest] 1 - [39m05/03/2023, 4:59:27 PM [31m ERROR[39m [38;5;3m[SmartInfoService] [39m[31mUnable to run image tagging pipeline: 87d3287a-e065-4285-bf53-9e89ece0af2b[39m
2023-05-03T14:59:27.937906644Z stderr at processTicksAndRejections (node:internal/process/task_queues:83:21)
2023-05-03T14:59:27.937862316Z stderr at endReadableNT (node:internal/streams/readable:1345:12)
2023-05-03T14:59:27.937819400Z stderr at IncomingMessage.emit (node:events:539:35)
2023-05-03T14:59:27.937770555Z stderr at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
2023-05-03T14:59:27.937716017Z stderr at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
2023-05-03T14:59:27.937663955Z stderr at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
2023-05-03T14:59:27.937524942Z stderr Error: Request failed with status code 500
2023-05-03T14:59:27.936665753Z stderr [31m[Nest] 1 - [39m05/03/2023, 4:59:27 PM [31m ERROR[39m [38;5;3m[SmartInfoService] [39m[31mUnable run object detection pipeline: faea0b72-fd88-4821-a399-56a9aac32ea3[39m
2023-05-03T14:59:27.931326466Z stderr at processTicksAndRejections (node:internal/process/task_queues:83:21)
2023-05-03T14:59:27.931255801Z stderr at endReadableNT (node:internal/streams/readable:1345:12)
2023-05-03T14:59:27.931206754Z stderr at IncomingMessage.emit (node:events:539:35)
2023-05-03T14:59:27.931155678Z stderr at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
2023-05-03T14:59:27.931100432Z stderr at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
2023-05-03T14:59:27.931029109Z stderr at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
2023-05-03T14:59:27.930917821Z stderr Error: Request failed with status code 500
2023-05-03T14:59:27.930445105Z stderr [31m[Nest] 1 - [39m05/03/2023, 4:59:27 PM [31m ERROR[39m [38;5;3m[SmartInfoService] [39m[31mUnable to run image tagging pipeline: faea0b72-fd88-4821-a399-56a9aac32ea3[39m
2023-05-03T14:59:27.605159927Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:27 PM [32m LOG[39m [38;5;3m[MetadataExtractionProcessor] [39m[32mReverse Geocoding Initialized[39m
2023-05-03T14:59:27.561095886Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:27 PM [32m LOG[39m [38;5;3m[ImmichMicroservice] [39m[32mRunning Immich Microservices in PRODUCTION environment - version 1.54.1 - Listening on port: 3002[39m
2023-05-03T14:59:27.549390423Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:27 PM [32m LOG[39m [38;5;3m[NestApplication] [39m[32mNest application successfully started[39m[38;5;3m +54ms[39m
2023-05-03T14:59:27.495161231Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:27 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mMicroservicesModule dependencies initialized[39m[38;5;3m +2ms[39m
2023-05-03T14:59:27.493669053Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:27 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mDomainModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:59:27.492208746Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:27 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mInfraModule dependencies initialized[39m[38;5;3m +24155ms[39m
2023-05-03T14:59:03.337670804Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:03 PM [32m LOG[39m [38;5;3m[MetadataExtractionProcessor] [39m[32mInitializing Reverse Geocoding[39m
2023-05-03T14:59:03.336622301Z stdout [33m[Nest] 1 - [39m05/03/2023, 4:59:03 PM [33m WARN[39m [38;5;3m[MetadataExtractionProcessor] [39m[33mReverse geocoding is enabled[39m
2023-05-03T14:59:03.332281469Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:03 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mTypeOrmModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:59:03.331194226Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:03 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mTypeOrmModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:59:03.329548493Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:03 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mTypeOrmCoreModule dependencies initialized[39m[38;5;3m +403ms[39m
2023-05-03T14:59:02.926440915Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:02 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mBullModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:59:02.925581035Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:02 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mBullModule dependencies initialized[39m[38;5;3m +2ms[39m
2023-05-03T14:59:02.923935714Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:02 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mConfigModule dependencies initialized[39m[38;5;3m +23ms[39m
2023-05-03T14:59:02.900961017Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:02 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mDiscoveryModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:59:02.899821223Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:02 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mConfigHostModule dependencies initialized[39m[38;5;3m +3ms[39m
2023-05-03T14:59:02.897050889Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:02 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mBullModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:59:02.895754215Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:02 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mTypeOrmModule dependencies initialized[39m[38;5;3m +178ms[39m
2023-05-03T14:59:02.754014924Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:59:02 PM [32m LOG[39m [38;5;3m[NestFactory] [39m[32mStarting Nest application...[39m
2023-05-03T14:44:39.596168171Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[MetadataExtractionProcessor] [39m[32mInitializing Reverse Geocoding[39m
2023-05-03T14:44:39.595255687Z stdout [33m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [33m WARN[39m [38;5;3m[MetadataExtractionProcessor] [39m[33mReverse geocoding is enabled[39m
2023-05-03T14:44:39.591182016Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mTypeOrmModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:44:39.590106772Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mTypeOrmModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:44:39.588495526Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mTypeOrmCoreModule dependencies initialized[39m[38;5;3m +434ms[39m
2023-05-03T14:44:39.154129097Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mBullModule dependencies initialized[39m[38;5;3m +2ms[39m
2023-05-03T14:44:39.152920300Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mBullModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:44:39.151586521Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mConfigModule dependencies initialized[39m[38;5;3m +23ms[39m
2023-05-03T14:44:39.128612098Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mDiscoveryModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:44:39.127444569Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mConfigHostModule dependencies initialized[39m[38;5;3m +3ms[39m
2023-05-03T14:44:39.124721140Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mBullModule dependencies initialized[39m[38;5;3m +1ms[39m
2023-05-03T14:44:39.123410832Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:39 PM [32m LOG[39m [38;5;3m[InstanceLoader] [39m[32mTypeOrmModule dependencies initialized[39m[38;5;3m +168ms[39m
2023-05-03T14:44:38.992375067Z stdout [32m[Nest] 1 - [39m05/03/2023, 4:44:38 PM [32m LOG[39m [38;5;3m[NestFactory] [39m[32mStarting Nest application...[39m
This is also a case in https://github.com/immich-app/immich/issues/2367 . Can we do something to handle this? Is the issue reported in the upstream repo? Does someone understand the reason for this bug?
@sakowicz The plan is to fork and rewrite the library to properly handle error so it does not crash
Is there a way to prevent immich from trying to Geocode ? I am struggling with this issue and my uploads of old photos always stops half way. I have tried uploading 3 or 4 times now and would be great if I can run immich without tagging location.
Same here did "DISABLE_REVERSE_GEOCODING=true" for now and it seams to be a work around.
