dawarich icon indicating copy to clipboard operation
dawarich copied to clipboard
verified publisher icon Freika

Problem importing data from Immich

Open daggr8 opened this issue 1 year ago • 1 comments

Describe the bug When using the "Import Immich Data" nothing is imported.

Version 0.14.3 Immich version 1.115

To Reproduce Go to /imports and click "import Immich Data"

Expected behavior Data imported

Logs

If applicable, add logs from containers `dawarich_app` and `dawarich_sidekiq` to help explain your problem.
Logs from dawarich_app
18:58:27 web.1  | D, [2024-09-22T18:58:27.283999 #86] DEBUG -- :   ↳ app/views/shared/_navbar.html.erb:69
18:58:27 web.1  | I, [2024-09-22T18:58:27.285813 #86]  INFO -- : {"method":"GET","path":"/settings","format":"html","controller":"SettingsController","action":"index","status":200,"allocations":13395,"duration":68.93,"view":59.22,"db":1.13}

Logs from dawarich_sidekiq

D, [2024-09-22T18:58:39.269194 #81] DEBUG -- :   ↳ app/jobs/import_immich_geodata_job.rb:7:in `perform'
E, [2024-09-22T18:58:52.460263 #81] ERROR -- : Error performing ImportImmichGeodataJob (Job ID: 57922463-1189-4e0a-93f2-1352e6381e4b) from Sidekiq(imports) in 30824.76ms: NoMethodError (undefined method `[]' for nil):
/var/app/app/services/immich/import_geodata.rb:85:in `file_name'
/var/app/app/services/immich/import_geodata.rb:18:in `call'
/var/app/app/jobs/import_immich_geodata_job.rb:9:in `perform'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:68:in `block in _perform_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/i18n-1.14.6/lib/i18n.rb:353:in `with_locale'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/core_ext/time/zones.rb:65:in `use_zone'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/timezones.rb:9:in `block (2 levels) in <module:Timezones>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:141:in `run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:67:in `_perform_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:32:in `_perform_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:51:in `perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:26:in `block in perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.1/lib/active_record/railties/job_runtime.rb:13:in `block in instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:40:in `block in instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/notifications.rb:210:in `block in instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/notifications.rb:210:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:39:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.1/lib/active_record/railties/job_runtime.rb:11:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:26:in `perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/logging.rb:32:in `block in perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/logging.rb:41:in `tag_logger'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/logging.rb:32:in `perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:29:in `block in execute'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/railtie.rb:79:in `block (4 levels) in <class:Railtie>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:77:in `block in wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:74:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/railtie.rb:78:in `block (3 levels) in <class:Railtie>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:141:in `run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:27:in `execute'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:70:in `perform'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:220:in `execute_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:185:in `block (4 levels) in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:180:in `traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job/interrupt_handler.rb:9:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/metrics/tracking.rb:26:in `track'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/metrics/tracking.rb:134:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:173:in `invoke'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:184:in `block (3 levels) in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:145:in `block (6 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_retry.rb:118:in `local'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:144:in `block (5 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/rails.rb:16:in `block in call'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:77:in `block in wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/execution_wrapper.rb:91:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:74:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/rails.rb:15:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:139:in `block (4 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:281:in `stats'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:134:in `block (3 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_logger.rb:23:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:133:in `block (2 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_retry.rb:85:in `global'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:132:in `block in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_logger.rb:50:in `prepare'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:131:in `dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:183:in `block (2 levels) in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:182:in `handle_interrupt'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:182:in `block in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:181:in `handle_interrupt'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:181:in `process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:86:in `process_one'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:76:in `run'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/component.rb:10:in `watchdog'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/component.rb:19:in `block in safe_thread'
2024-09-22T18:58:52.463Z pid=81 tid=ilt class=ImportImmichGeodataJob jid=ccbc8efd465953083517131b elapsed=30.849 INFO: fail
2024-09-22T18:58:52.463Z pid=81 tid=ilt WARN: {"context":"Job raised exception","job":{"retry":true,"queue":"imports","wrapped":"ImportImmichGeodataJob","args":[{"job_class":"ImportImmichGeodataJob","job_id":"57922463-1189-4e0a-93f2-1352e6381e4b","provider_job_id":null,"queue_name":"imports","priority":null,"arguments":[1],"executions":0,"exception_executions":{},"locale":"en","timezone":"Europe/Stockholm","enqueued_at":"2024-09-22T18:58:21.613987308Z","scheduled_at":null}],"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","jid":"ccbc8efd465953083517131b","created_at":1727031501.6141045,"enqueued_at":1727031501.6141832}}
2024-09-22T18:58:52.463Z pid=81 tid=ilt WARN: NoMethodError: undefined method `[]' for nil
2024-09-22T18:58:52.464Z pid=81 tid=ilt WARN: app/services/immich/import_geodata.rb:85:in `file_name'
app/services/immich/import_geodata.rb:18:in `call'
app/jobs/import_immich_geodata_job.rb:9:in `perform'
E, [2024-09-22T18:59:10.418145 #81] ERROR -- : Error performing ImportImmichGeodataJob (Job ID: a75fd068-a18f-4cb1-a609-31ec2276ab90) from Sidekiq(imports) in 31156.93ms: NoMethodError (undefined method `[]' for nil):
/var/app/app/services/immich/import_geodata.rb:85:in `file_name'
/var/app/app/services/immich/import_geodata.rb:18:in `call'
/var/app/app/jobs/import_immich_geodata_job.rb:9:in `perform'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:68:in `block in _perform_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/i18n-1.14.6/lib/i18n.rb:353:in `with_locale'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/core_ext/time/zones.rb:65:in `use_zone'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/timezones.rb:9:in `block (2 levels) in <module:Timezones>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:141:in `run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:67:in `_perform_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:32:in `_perform_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:51:in `perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:26:in `block in perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.1/lib/active_record/railties/job_runtime.rb:13:in `block in instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:40:in `block in instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/notifications.rb:210:in `block in instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/notifications.rb:210:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:39:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.1/lib/active_record/railties/job_runtime.rb:11:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:26:in `perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/logging.rb:32:in `block in perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/logging.rb:41:in `tag_logger'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/logging.rb:32:in `perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:29:in `block in execute'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/railtie.rb:79:in `block (4 levels) in <class:Railtie>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:77:in `block in wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:74:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/railtie.rb:78:in `block (3 levels) in <class:Railtie>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:141:in `run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:27:in `execute'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:70:in `perform'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:220:in `execute_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:185:in `block (4 levels) in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:180:in `traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job/interrupt_handler.rb:9:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/metrics/tracking.rb:26:in `track'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/metrics/tracking.rb:134:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:173:in `invoke'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:184:in `block (3 levels) in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:145:in `block (6 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_retry.rb:118:in `local'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:144:in `block (5 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/rails.rb:16:in `block in call'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:77:in `block in wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/execution_wrapper.rb:91:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:74:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/rails.rb:15:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:139:in `block (4 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:281:in `stats'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:134:in `block (3 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_logger.rb:23:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:133:in `block (2 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_retry.rb:85:in `global'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:132:in `block in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_logger.rb:50:in `prepare'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:131:in `dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:183:in `block (2 levels) in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:182:in `handle_interrupt'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:182:in `block in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:181:in `handle_interrupt'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:181:in `process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:86:in `process_one'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:76:in `run'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/component.rb:10:in `watchdog'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/component.rb:19:in `block in safe_thread'
2024-09-22T18:59:10.421Z pid=81 tid=hjp class=ImportImmichGeodataJob jid=0ff41d27f12370d7f95c606f elapsed=31.177 INFO: fail
2024-09-22T18:59:10.421Z pid=81 tid=hjp WARN: {"context":"Job raised exception","job":{"retry":true,"queue":"imports","wrapped":"ImportImmichGeodataJob","args":[{"job_class":"ImportImmichGeodataJob","job_id":"a75fd068-a18f-4cb1-a609-31ec2276ab90","provider_job_id":null,"queue_name":"imports","priority":null,"arguments":[1],"executions":0,"exception_executions":{},"locale":"en","timezone":"Europe/London","enqueued_at":"2024-09-22T17:30:45.656133959Z","scheduled_at":null}],"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","jid":"0ff41d27f12370d7f95c606f","created_at":1727026245.6562474,"enqueued_at":1727031519.2434938,"error_message":"undefined method `[]' for nil","error_class":"NoMethodError","failed_at":1727026276.2421787,"retry_count":7,"retried_at":1727029035.7479994}}
2024-09-22T18:59:10.424Z pid=81 tid=hjp WARN: NoMethodError: undefined method `[]' for nil
2024-09-22T18:59:10.425Z pid=81 tid=hjp WARN: app/services/immich/import_geodata.rb:85:in `file_name'
app/services/immich/import_geodata.rb:18:in `call'
app/jobs/import_immich_geodata_job.rb:9:in `perform'
2024-09-22T18:59:18.489Z pid=81 tid=j61 class=ImportImmichGeodataJob jid=ccbc8efd465953083517131b INFO: start
I, [2024-09-22T18:59:18.503214 #81]  INFO -- : Performing ImportImmichGeodataJob (Job ID: 57922463-1189-4e0a-93f2-1352e6381e4b) from Sidekiq(imports) enqueued at 2024-09-22T18:58:21.613987308Z with arguments: 1
D, [2024-09-22T18:59:18.509731 #81] DEBUG -- :   User Load (0.6ms)  SELECT "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]
D, [2024-09-22T18:59:18.510722 #81] DEBUG -- :   ↳ app/jobs/import_immich_geodata_job.rb:7:in `perform'

daggr8 avatar Sep 22 '24 19:09 daggr8

Mine worked... I think. though it seems low on the number of data points, and there is not status updates.

i am expecting much more than 26k datapoints. How do i check that it has worked?

palitu avatar Oct 05 '24 22:10 palitu

Same issue here with the NoMethodError; Immich version 1.117

Millio345 avatar Oct 14 '24 06:10 Millio345

Mine are imported but the reverse_geocoding doesn't work. I'm using Immich v1.118.2 I get Sidekiq(reverse_geocoding) in 10.41ms: NoMethodError (undefined method 'geodata' for nil)

2024-10-22T06:13:59.787114586Z E, [2024-10-22T06:13:59.786763 #115] ERROR -- : Error performing ReverseGeocodingJob (Job ID: 9c1b033c-0605-4b2f-a987-0b706510262a) from Sidekiq(reverse_geocoding) in 10.41ms: NoMethodError (undefined method `geodata' for nil):
2024-10-22T06:13:59.787137542Z /var/app/app/services/reverse_geocoding/points/fetch_data.rb:24:in `reverse_geocoded?'
2024-10-22T06:13:59.787150136Z /var/app/app/services/reverse_geocoding/points/fetch_data.rb:13:in `call'
2024-10-22T06:13:59.787162029Z /var/app/app/jobs/reverse_geocoding_job.rb:9:in `perform'

It would be quite helpful if we could get the payload that is causing these errors but for this we need to manually trigger the job and use a breakpoint.

There's a section on the docs that might be helpful: https://dawarich.app/docs/FAQ

guicattani avatar Oct 22 '24 07:10 guicattani

@guicattani

Sidekiq(reverse_geocoding) in 10.41ms: NoMethodError (undefined method 'geodata' for nil)

this means that the point, that was attempted to be reverse geocoded, was not found in the database. In this case, there would be nothing to log because there is no point in the first place.

Freika avatar Oct 22 '24 08:10 Freika

I have the same problem with an import from Immich v1.118.2

E, [2024-10-22T19:55:26.665714 #104] ERROR -- : Error performing Import::ImmichGeodataJob (Job ID: 2b075166-2d61-4ffb-8551-92fbc6a79c65) from Sidekiq(imports) in 34864.38ms: NoMethodError (undefined method `[]' for nil):
/var/app/app/services/immich/import_geodata.rb:85:in `file_name'
/var/app/app/services/immich/import_geodata.rb:18:in `call'
/var/app/app/jobs/import/immich_geodata_job.rb:10:in `perform'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/execution.rb:68:in `block in _perform_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/i18n-1.14.6/lib/i18n.rb:353:in `with_locale'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/core_ext/time/zones.rb:65:in `use_zone'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/timezones.rb:9:in `block (2 levels) in <module:Timezones>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:141:in `run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/execution.rb:67:in `_perform_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/instrumentation.rb:32:in `_perform_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/execution.rb:51:in `perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/instrumentation.rb:26:in `block in perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.1.1/lib/active_record/railties/job_runtime.rb:13:in `block in instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/instrumentation.rb:40:in `block in instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/notifications.rb:210:in `block in instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/notifications.rb:210:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/instrumentation.rb:39:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.1.1/lib/active_record/railties/job_runtime.rb:11:in `instrument'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/instrumentation.rb:26:in `perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/logging.rb:32:in `block in perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/logging.rb:41:in `tag_logger'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/logging.rb:32:in `perform_now'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/execution.rb:29:in `block in execute'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/railtie.rb:79:in `block (4 levels) in <class:Railtie>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/reloader.rb:77:in `block in wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/reloader.rb:74:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/railtie.rb:78:in `block (3 levels) in <class:Railtie>'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:141:in `run_callbacks'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/execution.rb:27:in `execute'
/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:70:in `perform'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:220:in `execute_job'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:185:in `block (4 levels) in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:180:in `traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job/interrupt_handler.rb:9:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/metrics/tracking.rb:26:in `track'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/metrics/tracking.rb:134:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:173:in `invoke'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:184:in `block (3 levels) in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:145:in `block (6 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_retry.rb:118:in `local'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:144:in `block (5 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/rails.rb:16:in `block in call'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/reloader.rb:77:in `block in wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/execution_wrapper.rb:91:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/reloader.rb:74:in `wrap'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/rails.rb:15:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:139:in `block (4 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:281:in `stats'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:134:in `block (3 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_logger.rb:23:in `call'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:133:in `block (2 levels) in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_retry.rb:85:in `global'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:132:in `block in dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_logger.rb:50:in `prepare'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:131:in `dispatch'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:183:in `block (2 levels) in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:182:in `handle_interrupt'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:182:in `block in process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:181:in `handle_interrupt'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:181:in `process'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:86:in `process_one'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:76:in `run'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/component.rb:10:in `watchdog'
/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/component.rb:19:in `block in safe_thread'
2024-10-22T19:55:26.666Z pid=104 tid=hl4 class=Import::ImmichGeodataJob jid=8f9a13a6ba4e2b9597cb4a66 elapsed=34.873 INFO: fail
2024-10-22T19:55:26.666Z pid=104 tid=hl4 WARN: {"context":"Job raised exception","job":{"retry":false,"queue":"imports","wrapped":"Import::ImmichGeodataJob","args":[{"job_class":"Import::ImmichGeodataJob","job_id":"2b075166-2d61-4ffb-8551-92fbc6a79c65","provider_job_id":null,"queue_name":"imports","priority":null,"arguments":[1],"executions":0,"exception_executions":{},"locale":"en","timezone":"Europe/Amsterdam","enqueued_at":"2024-10-22T19:54:51.792773869Z","scheduled_at":null}],"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","jid":"8f9a13a6ba4e2b9597cb4a66","created_at":1729626891.7928104,"enqueued_at":1729626891.792841}}
2024-10-22T19:55:26.666Z pid=104 tid=hl4 WARN: NoMethodError: undefined method `[]' for nil
2024-10-22T19:55:26.666Z pid=104 tid=hl4 WARN: app/services/immich/import_geodata.rb:85:in `file_name'

ilijamt avatar Oct 22 '24 19:10 ilijamt

@daggr8 @ilijamt @Millio345 in the 0.15.11 I introduced writing the response received from Immich to the tmp/imports/immich_raw_data_CURRENT_TIME_USER_EMAIL.json file, which you can then send me to debug your issue.

⚠️⚠️⚠️ The file contains structured EXIF info on the images from Immich, which might include sensitive data, such as username or file name (along with obviously necessary coordinates and timestamps). I'm only interested in the data structure, so feel free to anonymize the values. Although I can't advise on how exactly you can anonymize the data, you need to know what you might expose.

Example of JSON for a single image produced by Immich:

{"id"=>"b3f77c49-5335-4ad2-9032-afe04b8843f2",
 "deviceAssetId"=>"D04881DA-F413-494D-80CB-8B4CBC8BD37D.jpg-131770",
 "ownerId"=>"f579f328-c355-438c-a82c-fe3390bd5f08",
 "deviceId"=>"CLI",
 "libraryId"=>nil,
 "type"=>"IMAGE",
 "originalPath"=>"upload/library/admin/2001/2001-01-01/D04881DA-F413-494D-80CB-8B4CBC8BD37D.jpg",
 "originalFileName"=>"D04881DA-F413-494D-80CB-8B4CBC8BD37D.jpg",
 "originalMimeType"=>"image/jpeg",
 "thumbhash"=>"JPgVHAR394iniHeHZ3h4iOD4Bg==",
 "fileCreatedAt"=>"2000-12-31T23:00:00.000Z",
 "fileModifiedAt"=>"2001-01-01T00:00:00.000Z",
 "localDateTime"=>"2001-01-01T00:00:00.000Z",
 "updatedAt"=>"2024-05-21T14:39:22.205Z",
 "isFavorite"=>false,
 "isArchived"=>false,
 "isTrashed"=>false,
 "duration"=>"0:00:00.00000",
 "exifInfo"=>
  {"make"=>nil,
   "model"=>nil,
   "exifImageWidth"=>1024,
   "exifImageHeight"=>1820,
   "fileSizeInByte"=>131770,
   "orientation"=>"1",
   "dateTimeOriginal"=>"2000-12-31T23:00:00.000Z",
   "modifyDate"=>"2000-12-31T23:00:00.000Z",
   "timeZone"=>"Europe/Berlin",
   "lensModel"=>nil,
   "fNumber"=>nil,
   "focalLength"=>nil,
   "iso"=>nil,
   "exposureTime"=>nil,
   "latitude"=>11.111,
   "longitude"=>22.222,
   "city"=>"Hohenschwangau",
   "state"=>"Swabia, Bavaria",
   "country"=>"Germany",
   "description"=>"",
   "projectionType"=>nil,
   "rating"=>nil},
 "livePhotoVideoId"=>nil,
 "people"=>[],
 "checksum"=>"7PGkxQcyGjRNy82gV9vOKcWZy94=",
 "stack"=>nil,
 "isOffline"=>false,
 "hasMetadata"=>true,
 "duplicateId"=>nil,
 "resized"=>true}

If you could send me your file(s) in Discord, I could work on the fix. My importing process works without any errors, so I need an example of a payload that causes errors to be raised.

Freika avatar Oct 29 '24 11:10 Freika

Hi,

For me the .json file doesn't exist.

2024-10-29T12:00:53.842Z pid=88 tid=hlg class=Import::ImmichGeodataJob jid=414c894469ad708f0b2ea3b0 elapsed=33.981 INFO: fail
2024-10-29T12:00:53.843Z pid=88 tid=hlg WARN: {"context":"Job raised exception","job":{"retry":false,"queue":"imports","wrapped":"Import::ImmichGeodataJob","args":[{"job_class":"Import::ImmichGeodataJob","job_id":"e0601d58-4daf-47be-a8b7-2c60bd6f7d59","provider_job_id":null,"queue_name":"imports","priority":null,"arguments":[1],"executions":0,"exception_executions":{},"locale":"en","timezone":"Europe/Zone","enqueued_at":"2024-10-29T12:00:19.860914115Z","scheduled_at":null}],"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","jid":"414c894469ad708f0b2ea3b0","created_at":1730203219.86115,"enqueued_at":1730203219.8612497}}
2024-10-29T12:00:53.843Z pid=88 tid=hlg WARN: Errno::ENOENT: No such file or directory @ rb_sysopen - tmp/imports/immich_raw_data_2024-10-29 10:00:53 +0000_myemail.json
2024-10-29T12:00:53.844Z pid=88 tid=hlg WARN: app/services/immich/import_geodata.rb:81:in `initialize'

daggr8 avatar Oct 29 '24 12:10 daggr8

Immich import is also failing for me. Immich import has worked in the past from the same Immich server, so either I have a new bad photo or something changed in Immich or Dawarich recently.

I notice the OP had this error, as did a subsequent poster:

E, [2024-09-22T18:58:52.460263 #81] ERROR -- : Error performing ImportImmichGeodataJob (Job ID: 57922463-1189-4e0a-93f2-1352e6381e4b) from Sidekiq(imports) in 30824.76ms: NoMethodError (undefined method []' for nil):`

But now others and myself are seeing this instead: E, [2024-11-04T16:18:36.195705 #116] ERROR -- : Error performing Import::ImmichGeodataJob (Job ID: 1ef8c659-3cdd-41c4-a92c-081f750764d0) from Sidekiq(imports) in 41241.91ms: Errno::ENOENT (No such file or directory @ rb_sysopen - tmp/imports/immich_raw_data_2024-11-04 10:18:36 [email protected]):

chrisl8 avatar Nov 04 '24 16:11 chrisl8

It appears that the Sidekiq container needs to have a /var/app/tmp/imports folder.

I fixed the issue by doing this:

docker exec -it dawarich_sidekiq sh
mkdir -p /var/app/tmp/imports
exit

And then run the import again.

This fix will only last until you restart the container(s), but it should get the import to work.

I'm not sure why we are missing that folder, perhaps because we didn't set up a "watch" folder and so it doesn't get auto-created?

Seems like the process should create the import folder if/when it needs it, but it does not.

chrisl8 avatar Nov 04 '24 16:11 chrisl8

I followed the tip above and i get a json containing this. I tried it multiple times with the same result.

[[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]]

daggr8 avatar Nov 05 '24 18:11 daggr8

I followed the tip above and i get a json containing this. I tried it multiple times with the same result.

This one is really interesting, it means Dawarich receives a list of empty arrays so there are no points to get data from. I think I'll change approach to writing the debugging file and will log whole response from Immich, so we could investigate it further

Freika avatar Nov 07 '24 12:11 Freika

It appears that the Sidekiq container needs to have a /var/app/tmp/imports folder.

In the default docker-compose.yml there are these lines:

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    volumes:
      - gem_cache:/usr/local/bundle/gems_app
      - public:/var/app/public
      - watched:/var/app/tmp/imports/watched

last one assumes your instance should have this directory, but I guess if you didn't use watched functionality, Dawarich should create it on its own. I'll look into it too

Freika avatar Nov 07 '24 12:11 Freika

Here is a script for you guys to see what data is being returned by your Immich instance:

#!/bin/bash

# Set your API key and base URL here
IMMICH_API_KEY="YOUR_API_KEY"
IMMICH_API_BASE_URL="http://immich.instance.com/api"

# Output file
OUTPUT_FILE="immich_data.json"

# Temporary file for storing intermediate responses
TEMP_FILE="immich_data_temp.json"
echo "[]" > $TEMP_FILE  # Initialize as an empty JSON array

# Calculate total requests for progress tracking
START_YEAR=1970
END_YEAR=$(date +"%Y")
TOTAL_REQUESTS=$(( (END_YEAR - START_YEAR + 1) * 12 ))

# Initialize counters
current_request=0

# Loop through each year and month, retrieve data, and append to TEMP_FILE
for year in $(seq $START_YEAR $END_YEAR); do
  for month in $(seq -w 1 12); do
    # Construct the URL
    url="${IMMICH_API_BASE_URL}/timeline/bucket?size=MONTH&timeBucket=${year}-${month}-01"

    # Send the request and get the response
    response=$(curl -s -H "x-api-key: $IMMICH_API_KEY" -H "accept: application/json" "$url")

    # Check if the response is valid JSON and append it to TEMP_FILE
    if echo "$response" | jq empty 2>/dev/null; then
      # Add the response to TEMP_FILE JSON array
      jq ". += [$response]" $TEMP_FILE > tmp.json && mv tmp.json $TEMP_FILE
    else
      echo "Failed to retrieve data for ${year}-${month}"
    fi

    # Increment the counter and update progress bar
    current_request=$((current_request + 1))
    progress=$((current_request * 100 / TOTAL_REQUESTS))
    printf "\rProgress: [%-50s] %d%%" $(printf '#%.0s' $(seq 1 $((progress / 2)))) $progress
  done
done

# Move the final JSON array from TEMP_FILE to OUTPUT_FILE
mv $TEMP_FILE $OUTPUT_FILE

# Complete the progress bar and show final message
echo -e "\nData collection complete. Output written to $OUTPUT_FILE."

Save it as immich_geodata.sh, chown +x immich_geodata.sh it and then run it: ./immich_geodata.sh

The result will be written into immich_data.json file.

Then feel free to send me the results via DM in discord and I'll be able to debug it. Hope this helps!

Freika avatar Nov 08 '24 15:11 Freika

I don't use Discord unfortunately. I hope it's okay to send it here. The immich_data.json file doesn't contain any data, just hundreds of rows of

  [],
  [],
  [],

daggr8 avatar Nov 08 '24 15:11 daggr8

I don't use Discord unfortunately. I hope it's okay to send it here.

The immich_data.json file doesn't contain any data, just hundreds of rows of


  [],

  [],

  [],

In the immich interface, do your photos have info on geodata? Are they shown on immich map?

Freika avatar Nov 08 '24 15:11 Freika

Also, the resulting file might contain a lot of empty arrays in the beginning and then the actual data is way below them, so scrolling might help :)

Freika avatar Nov 08 '24 15:11 Freika

I don't use Discord unfortunately. I hope it's okay to send it here. The immich_data.json file doesn't contain any data, just hundreds of rows of

  [],

  [],

  [],

In the immich interface, do your photos have info on geodata? Are they shown on immich map?

Yepp they do.

And the file is empty, no data besides hundreds of rows of [],. I've double checked :)

daggr8 avatar Nov 08 '24 15:11 daggr8

Hmm so curl -s -H "x-api-key: your_api_key_here" -H "accept: application/json" "https://your_base_url_here/api/timeline/bucket?size=MONTH&timeBucket=2024-10-01" -o immich_data.json will return you an empty result? Replace year and month to ones you know for sure have pictures

Freika avatar Nov 08 '24 15:11 Freika

curl -s -H "x-api-key: your_api_key_here" -H "accept: application/json" "https://your_base_url_here/api/timeline/bucket?size=MONTH&timeBucket=2024-10-01" -o immich_data.json

Same problem and yes, empty result, even though there is geodata in the photos for the day specified.

tbelway avatar Nov 08 '24 15:11 tbelway

Using baseurl/api/timeline returns []%

Using baseurl/timeline returns a page

I've tried multiple dates.

I can add that it takes a minute or two for the script to run and while it does the CPU on my Immich server is spiking, so it definitely talks to the API. I also just upgraded to Immich 1.120.1 and ran it again. Same result.

daggr8 avatar Nov 08 '24 15:11 daggr8

Then I'm afraid it's something with Immich, because if there is data in your instance, it should be returned, but it's not :(

Freika avatar Nov 08 '24 15:11 Freika

Then I'm afraid it's something with Immich, because if there is data in your instance, it should be returned, but it's not :(

I'm not confident that's the only problem, i think there are two:

WARN: Socket::ResolutionError: Failed to open TCP connection to ${REDACTED_URL}:443 (getaddrinfo: Try again)

If i add a hosts entry in the sidekiq container and use the local IP and port then it's able to connect and then subsequently, but it fails over https. I think your code may not be able to handle https?

The subsequent error is the one predominantly experienced in this issues ticket:

WARN: NoMethodError: undefined method []' for nil`

I created an issue within the immich github for this: https://github.com/immich-app/immich/issues/14084

tbelway avatar Nov 11 '24 19:11 tbelway

This tool should not use the timeline endpoints to extract/download gps data from immich. The timeline endpoints are optimized for specific use cases, like hiding archived photos, showing only trashed photos, etc., and are subject to change. Please use the "search metadata" endpoint for bulk operations like this.

https://immich.app/docs/api/search-metadata

jrasm91 avatar Nov 14 '24 18:11 jrasm91

In 0.16.5 I reworked the integration to use POST /api/search/metadata to get geodata, if you're experiencing problems with immich integration, update and try to import the data again

Freika avatar Nov 18 '24 15:11 Freika

At first run i got an error that /var/app/tmp/imports didn't exist. I manually created it and ran the job again. So far so good, there's a 50MB .json file and the log is going crazy.

However the "Jobs in queue" is at 0.

I'll update when it's finished.

daggr8 avatar Nov 21 '24 07:11 daggr8

this seems to be working! Thank you :)

tbelway avatar Nov 22 '24 02:11 tbelway

Seems to work for me as well. Thank you :)

daggr8 avatar Nov 23 '24 20:11 daggr8