dawarich icon indicating copy to clipboard operation
dawarich copied to clipboard
verified publisher icon Freika

Parsing GeoJSON without timestamps

Open dmitrym0 opened this issue 1 year ago • 7 comments

Describe the bug

Dawarich's GeoJSON parsing expects a timestamp to be included for every feature.

Some GeoJSON exports (see wanderin.gs) do not include them.

This is a valid GeoJSON:

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {},
      "geometry": {
        "coordinates": [
          -98.94065955738432,
          39.12986446576923
        ],
        "type": "Point"
      }
    },
  ]
}

but Dawarich will throw an exception here because it expects timestamp to be included.

I overrode this behaviour thusly, so something is returned.

  def timestamp(feature)
    return Time.zone.at(feature[3]) if feature.is_a?(Array)

    value = feature.dig(:properties, :timestamp) || feature.dig(:geometry, :coordinates, 3)
    Time.zone.at(Time.now)
  end

I figure this likely breaks line segment drawing because some points are out of order. Happy to work on a fix if you have thoughts on how to fix this better - if timestamps are not available.

Version

0.13.5

To Reproduce

  1. Create a valid geojson file (see above).
  2. Create an import, specify the json file.
  3. Observe an exception in sidekiq log.

Expected behavior

Import timestamp-less GeoJSON. For ordering we can perhaps use the ordering from the json file?

Logs

dawarich_sidekiq           | D, [2024-09-15T02:35:47.443707 #78] DEBUG -- :   Notification Create (5.2ms)  INSERT INTO "notifications" ("title", "content", "user_id", "kind", "read_at", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7) RETURNING "id"  [["title", "Import failed"], ["content", "Import \"2.json\" failed: can't convert NilClass into an exact number, stacktrace: <internal:timev>:286:in `at'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/core_ext/time/calculations.rb:52:in `at_with_coercion'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/values/time_zone.rb:382:in `at'\n/var/app/app/services/geojson/params.rb:88:in `timestamp'\n/var/app/app/services/geojson/params.rb:40:in `build_point'\n/var/app/app/services/geojson/params.rb:22:in `process_feature'\n/var/app/app/services/geojson/params.rb:31:in `block in process_feature_collection'\n/var/app/app/services/geojson/params.rb:31:in `map'\n/var/app/app/services/geojson/params.rb:31:in `process_feature_collection'\n/var/app/app/services/geojson/params.rb:13:in `call'\n/var/app/app/services/geojson/import_parser.rb:13:in `call'\n/var/app/app/services/imports/create.rb:12:in `call'\n/var/app/app/models/import.rb:17:in `process!'\n/var/app/app/jobs/import_job.rb:10:in `perform'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:68:in `block in _perform_job'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'\n/var/app/vendor/bundle/ruby/3.3.0/gems/i18n-1.14.5/lib/i18n.rb:351:in `with_locale'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `instance_exec'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/core_ext/time/zones.rb:65:in `use_zone'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/timezones.rb:9:in `block (2 levels) in <module:Timezones>'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `instance_exec'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:141:in `run_callbacks'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:67:in `_perform_job'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:32:in `_perform_job'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:51:in `perform_now'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:26:in `block in perform_now'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.1/lib/active_record/railties/job_runtime.rb:13:in `block in instrument'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:40:in `block in instrument'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/notifications.rb:210:in `block in instrument'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/notifications/instrumenter.rb:58:in `instrument'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/notifications.rb:210:in `instrument'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:39:in `instrument'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.1/lib/active_record/railties/job_runtime.rb:11:in `instrument'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/instrumentation.rb:26:in `perform_now'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/logging.rb:32:in `block in perform_now'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/logging.rb:41:in `tag_logger'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/logging.rb:32:in `perform_now'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:29:in `block in execute'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/railtie.rb:79:in `block (4 levels) in <class:Railtie>'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:77:in `block in wrap'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:74:in `wrap'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/railtie.rb:78:in `block (3 levels) in <class:Railtie>'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `instance_exec'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/callbacks.rb:141:in `run_callbacks'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/execution.rb:27:in `execute'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activejob-7.2.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:70:in `perform'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:220:in `execute_job'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:185:in `block (4 levels) in process'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:180:in `traverse'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job/interrupt_handler.rb:9:in `call'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:182:in `traverse'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/metrics/tracking.rb:26:in `track'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/metrics/tracking.rb:134:in `call'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:182:in `traverse'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/middleware/chain.rb:173:in `invoke'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:184:in `block (3 levels) in process'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:145:in `block (6 levels) in dispatch'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_retry.rb:118:in `local'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:144:in `block (5 levels) in dispatch'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/rails.rb:16:in `block in call'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:77:in `block in wrap'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/execution_wrapper.rb:91:in `wrap'\n/var/app/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.1/lib/active_support/reloader.rb:74:in `wrap'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/rails.rb:15:in `call'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:139:in `block (4 levels) in dispatch'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:281:in `stats'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:134:in `block (3 levels) in dispatch'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_logger.rb:23:in `call'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:133:in `block (2 levels) in dispatch'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_retry.rb:85:in `global'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:132:in `block in dispatch'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/job_logger.rb:50:in `prepare'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:131:in `dispatch'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:183:in `block (2 levels) in process'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:182:in `handle_interrupt'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:182:in `block in process'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:181:in `handle_interrupt'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:181:in `process'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:86:in `process_one'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/processor.rb:76:in `run'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/component.rb:10:in `watchdog'\n/var/app/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.2/lib/sidekiq/component.rb:19:in `block in safe_thread'"], ["user_id", 1], ["kind", 2], ["read_at", nil], ["created_at", "2024-09-15 02:35:47.436838"], ["updated_at", "2024-09-15 02:35:47.436838"]]

dmitrym0 avatar Sep 15 '24 02:09 dmitrym0

I'd say, returning Time.now is not a valid behavior. Dawarich operates within 3 dimensions: two of them are latitude and longitude, and the third is time. Without time provided along with space, it's impossible to establish when a point was recorded, hence location history will make no sense, there is no history when there is no time. Even if a geojson record is valid per se.

Does wanderin.gs not return any timestamp in somehow otherwise named attribute within properties?

Freika avatar Sep 15 '24 18:09 Freika

Time.now is absolutely not right, I was just trying to get it to work.

Looking at the output json a little closer:

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "dur": 5235,
        "ts": 1516068164735,
        "acc": 1000,
        "age": 209325249702,
        "src": "w"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
       ]
      }
    },

I took a peek at the geojson rfc it doesn't look like there is any specific time stamp definition. I'll see if I can play with this locally a bit more. Seems like ts should be the timestamp, not sure what age and duration refer to.

dmitrym0 avatar Sep 18 '24 17:09 dmitrym0

@dmitrym0 ts looks like a timestamp, is it also a result of export from wanderings?

Freika avatar Sep 18 '24 18:09 Freika

@Freika yes it is.

dmitrym0 avatar Sep 18 '24 18:09 dmitrym0

@dmitrym0 I wonder why first example didn't have it 🤔 If we can be sure that ts will be included in all files, then I can implement support of this attribute. Too bad geojson rfc doesn't have description of at least optional timestamp

Freika avatar Sep 18 '24 19:09 Freika

I'd say, returning Time.now is not a valid behavior. Dawarich operates within 3 dimensions: two of them are latitude and longitude, and the third is time. Without time provided along with space, it's impossible to establish when a point was recorded, hence location history will make no sense, there is no history when there is no time.

Perhaps it can be a two-step process? You can get the timezone from the lat/long combination, and then apply it to the timestamp. I used that approach to geocode the EXIF files, that store the time without a timezone.

I believe, it won't work only for that 1 ambiguous hour during the autumn DST change.

Cyberax avatar Sep 20 '24 21:09 Cyberax

Perhaps it can be a two-step process? You can get the timezone from the lat/long combination, and then apply it to the timestamp. I used that approach to geocode the EXIF files, that store the time without a timezone.

Great idea, would've never occurred to me. Time handling is a challenge!

dmitrym0 avatar Sep 20 '24 23:09 dmitrym0