Freika
undefined method 'y' for nil
OS & Hardware Quad Core x86 CPU 2.3 GHz 4 GB RAM Ubuntu 22.04
Version 0.25.4 & 0.25.3
Describe the bug After the 0.25.3 update I had the "undefined method 'y' for nil" bug. I followed the instructions from the releases page and got the map working again https://github.com/Freika/dawarich/releases/tag/0.25.2
Today, when checking the map the error suddenly popped up again without me having done anything on the system. I tried updating to 0.25.4, but that didn't fix the issue either.
I ran the steps from the FAQ https://dawarich.app/docs/FAQ/#how-to-enter-dawarich-console but those didn't help either
To Reproduce Steps to reproduce the behavior:
- Go to '/map' or login
- See error
Expected behavior See the map
Screenshots
Additional context If you need me to provide more info, just let me know :)
I am seeing the same issue. I have run the command to fix this several times. It consumes 9 gb of memory while running. It seemingly completes, but it still give me the error if I change date.
/var/app # bin/rails console
[dotenv] Set DATABASE_PORT
[dotenv] Loaded .env.development
Loading development environment (Rails 8.0.1)
[1] pry(main)> User.includes(:tracked_points).find_each do |user|
user.tracked_points.where(lonlat: nil).update_all('lonlat = ST_SetSRID(ST_MakePoint(longitude, latitude), 4326)')
[1] pry(main)> User.includes(:tracked_points).find_each do |user|
user.tracked_points.where(lonlat: nil).update_all('lonlat = ST_SetSRID(ST_MakePoint(longitude, latitude), 4326)')
D, [2025-04-08T21:34:25.420989 #33997] DEBUG -- : User Load (5.1ms) SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1 [["LIMIT", 1000]]
D, [2025-04-08T21:35:40.962038 #33997] DEBUG -- : Point Load (75009.3ms) SELECT "points".* FROM "points" WHERE "points"."user_id" = $1 [["user_id", 1]]
D, [2025-04-08T21:37:17.199766 #33997] DEBUG -- : Point Update All (31419.7ms) UPDATE "points" SET lonlat = ST_SetSRID(ST_MakePoint(longitude, latitude), 4326) WHERE "points"."user_id" = $1 AND "points"."lonlat" IS NULL [["user_id", 1]]
=> nil
[2] pry(main)>
I also get this on the "points" page
Showing /var/app/app/views/points/_point.html.erb where line #18 raised:
undefined method 'y' for nil
def lat
lonlat.y
end
private
Disclaimer.. I don´t exactly know the datamodel in dawarich, so I don´t know if the procedure below will trash the information in latitude and longitude. Use it at your own risk.
I browsed the data, and I had 137,000 records where longitude and latitude were nil, but raw_data contained the information.
I ran this script to populate latitude and longitude from the raw_data
user = User.find_by(email: '[email protected]')
user.tracked_points.where(latitude: nil, longitude: nil).find_each do |point|
raw = point.raw_data
next unless raw.is_a?(Hash) && raw["lat"].present? && raw["lon"].present?
point.latitude = raw["lat"]
point.longitude = raw["lon"]
point.save(validate: false)
end
And then I could run this to update the lonlat
rake data_cleanup:remove_duplicate_points
rake points:migrate_to_lonlat
some helper functions
user = User.find_by(email: '[email protected]')
user.tracked_points.where(latitude: nil, longitude: nil).size
user.tracked_points.where.not(latitude: nil, longitude: nil).where(lonlat: nil).size
@larspehrsson did it solve your issue? I can confirm the script you provided is a non-destructive one.
Running the scripts didn't help. I still get the error.
[7] pry(main)> user.tracked_points.where(latitude: nil, longitude: nil).size
D, [2025-04-09T23:47:22.019364 #3301] DEBUG -- : Point Count (1.1ms) SELECT COUNT(*) FROM "points" WHERE "points"."user_id" = $1 AND "points"."latitude" IS NULL AND "points"."longitude" IS NULL [["user_id", 1]]
=> 0
[8] pry(main)> user.tracked_points.where.not(latitude: nil, longitude: nil).where(lonlat: nil).size
D, [2025-04-09T23:47:28.184180 #3301] DEBUG -- : Point Count (561.5ms) SELECT COUNT(*) FROM "points" WHERE "points"."user_id" = $1 AND NOT ("points"."latitude" IS NULL AND "points"."longitude" IS NULL) AND "points"."lonlat" IS NULL [["user_id", 1]]
=> 1
It did solve my issue. Maybe you, @Freika, could change the rake points:migrate_to_lonlat script to do something similar if there are records where the latitude and longitude are nil but the value is in the raw_data.
If you have just one record left, @hartmark, I would have a look at it with user.tracked_points.where.not(latitude: nil, longitude: nil).where(lonlat: nil) and then probably just destroy it if points.size is still just 1.
something like:
points = user.tracked_points.where.not(latitude: nil, longitude: nil).where(lonlat: nil)
points.size
# if still just one, delete it.
points.each(&:destroy)
There is a similar solution here https://github.com/Freika/dawarich/issues/973#issuecomment-2749558448
It did solve my issue. Maybe you, @Freika, could change the rake points:migrate_to_lonlat script to do something similar if there are records where the latitude and longitude are nil but the value is in the raw_data.
If you have just one record left, @hartmark, I would have a look at it with
user.tracked_points.where.not(latitude: nil, longitude: nil).where(lonlat: nil)and then probably just destroy it if points.size is still just 1.something like:
points = user.tracked_points.where.not(latitude: nil, longitude: nil).where(lonlat: nil) points.size # if still just one, delete it. points.each(&:destroy)There is a similar solution here #973 (comment)
Cool, now it seems to be working properly for me.
I tried @larspehrsson's approach and it didn't work, unfortunately. Still getting the same error when going back to the date where the error occurs. It doesn't seem to happen to new or current data, though
I literally opened the database and edited the nulls. Fortunately, there were only a handful. In a graphic database app like Navicat or pgAdmin, you can see the raw values at the same time and copy/paste in the correct values. I would have written an update query, but for 11 it was just faster and easier to do it copy/paste. As soon as I did that, it worked great. No issues since.
For some reason the Rails commands didn't get all the null values. If you're good at writing rails commands, you can probably work around the issue and find a suitable way to copy all the values into lonlat.
@TheBig-O you copied values from raw_data, not longitude and `latitude? Were they empty?
Yes, they were both empty in all eleven rows. It was weird to me too. I have database backups that I can probably mount somehow if you need me to pull those records. (I think... would take some creative thinking to start a container around a backup database.) The raw data has full values though.
As soon as https://github.com/Freika/dawarich/releases/tag/0.25.5-rc0 is built, you can try using the freikin/dawarich:rc docker image to run Dawarich (just replace the image name in your compose file), it contains possible fix for the issue. The rake points:migrate_to_lonlat task now contains additional logic: it takes all points with empty longitude and latitude attrs and tries to fill them with appropriate data from the raw_data column.
The task might take some time and even be killed due to insufficient memory, but running it multiple times until it succeeds is a viable option.
As soon as https://github.com/Freika/dawarich/releases/tag/0.25.5-rc0 is built, you can try using the
freikin/dawarich:rcdocker image to run Dawarich (just replace the image name in your compose file), it contains possible fix for the issue. Therake points:migrate_to_lonlattask now contains additional logic: it takes all points with emptylongitudeandlatitudeattrs and tries to fill them with appropriate data from theraw_datacolumn.The task might take some time and even be killed due to insufficient memory, but running it multiple times until it succeeds is a viable option.
Not sure if its related or not but on the RC version I get the following error when running it: Docker in Unraid.
/var/app # bin/rake points:migrate_to_lonlat
[dotenv] Loaded .env.development
Updating points to use lonlat...
WARNING: database "dawarich" has no actual collation version, but a version was recorded
D, [2025-04-14T12:03:51.835487 #124] DEBUG -- : Point Load (6.0ms) SELECT "points"."id", "points"."longitude", "points"."latitude", "points"."raw_data" FROM "points" WHERE "points"."longitude" IS NULL AND "points"."latitude" IS NULL ORDER BY "points"."id" ASC LIMIT $1 [["LIMIT", 1000]]
D, [2025-04-14T12:03:51.838711 #124] DEBUG -- : ↳ lib/tasks/points.rake:10:in 'block (2 levels) in <main>'
rake aborted!
NoMethodError: undefined method '[]' for nil (NoMethodError)
if point.raw_data['activitySegment']['waypointPath']['waypoints'][0]
^^^^^^^^^^^^^^^^
/var/app/app/services/points/raw_data_lonlat_extractor.rb:21:in 'Points::RawDataLonlatExtractor#extract_lonlat'
/var/app/app/services/points/raw_data_lonlat_extractor.rb:9:in 'Points::RawDataLonlatExtractor#call'
/var/app/lib/tasks/points.rake:11:in 'block (3 levels) in <main>'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:88:in 'Array#each'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:88:in 'block in ActiveRecord::Batches#find_each'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:172:in 'block in ActiveRecord::Batches#find_in_batches'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:461:in 'block in ActiveRecord::Batches#batch_on_unloaded_relation'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:434:in 'ActiveRecord::Batches#batch_on_unloaded_relation'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:289:in 'ActiveRecord::Batches#in_batches'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:171:in 'ActiveRecord::Batches#find_in_batches'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:87:in 'ActiveRecord::Batches#find_each'
/var/app/lib/tasks/points.rake:10:in 'block (2 levels) in <main>'
Tasks: TOP => points:migrate_to_lonlat
(See full trace by running task with --trace)
/var/app #
This is on the sidekiq container:
/var/app # bin/rake points:migrate_to_lonlat
[dotenv] Loaded .env.development
Updating points to use lonlat...
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
WARNING: database "dawarich" has no actual collation version, but a version was recorded
D, [2025-04-14T12:06:26.179490 #1049] DEBUG -- : (6163.7ms) REINDEX TABLE points;
D, [2025-04-14T12:06:26.179819 #1049] DEBUG -- : ↳ lib/tasks/points.rake:8:in 'block (2 levels) in <main>'
D, [2025-04-14T12:06:26.180257 #1049] DEBUG -- : TRANSACTION (0.2ms) BEGIN
D, [2025-04-14T12:06:26.181799 #1049] DEBUG -- : ↳ lib/tasks/points.rake:11:in 'block (3 levels) in <main>'
D, [2025-04-14T12:06:26.218483 #1049] DEBUG -- : (18.4ms) ALTER TABLE points DISABLE TRIGGER ALL;
D, [2025-04-14T12:06:26.220710 #1049] DEBUG -- : ↳ lib/tasks/points.rake:11:in 'block (3 levels) in <main>'
D, [2025-04-14T12:06:26.335845 #1049] DEBUG -- : (110.4ms) UPDATE points
SET lonlat = ST_SetSRID(ST_MakePoint(longitude, latitude), 4326)::geography
WHERE lonlat IS NULL
AND longitude IS NOT NULL
AND latitude IS NOT NULL;
D, [2025-04-14T12:06:26.336190 #1049] DEBUG -- : ↳ lib/tasks/points.rake:14:in 'block (3 levels) in <main>'
D, [2025-04-14T12:06:26.339218 #1049] DEBUG -- : TRANSACTION (0.1ms) ROLLBACK
D, [2025-04-14T12:06:26.350995 #1049] DEBUG -- : ↳ lib/tasks/points.rake:10:in 'block (2 levels) in <main>'
rake aborted!
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_points_on_lonlat_timestamp_user_id" (ActiveRecord::RecordNotUnique)
DETAIL: Key (lonlat, "timestamp", user_id)=(0101000020E6100000FF78AF5A995821C0D099B4A9BA934340, 1744624450, 1) already exists.
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in 'PG::Connection#exec'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in 'ActiveRecord::ConnectionAdapters::PostgreSQL::DatabaseStatements#perform_query'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in 'block (2 levels) in ActiveRecord::ConnectionAdapters::DatabaseStatements#raw_execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:1011:in 'block in ActiveRecord::ConnectionAdapters::AbstractAdapter#with_raw_connection'
/var/app/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/concurrency/null_lock.rb:9:in 'ActiveSupport::Concurrency::NullLock#synchronize'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:983:in 'ActiveRecord::ConnectionAdapters::AbstractAdapter#with_raw_connection'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in 'block in ActiveRecord::ConnectionAdapters::DatabaseStatements#raw_execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/notifications/instrumenter.rb:58:in 'ActiveSupport::Notifications::Instrumenter#instrument'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:1129:in 'ActiveRecord::ConnectionAdapters::AbstractAdapter#log'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#raw_execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#internal_execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:137:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/query_cache.rb:27:in 'ActiveRecord::ConnectionAdapters::AbstractAdapter#execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:40:in 'ActiveRecord::ConnectionAdapters::PostgreSQL::DatabaseStatements#execute'
/var/app/lib/tasks/points.rake:14:in 'block (3 levels) in <main>'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:626:in 'block in ActiveRecord::ConnectionAdapters::TransactionManager#within_new_transaction'
/var/app/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/concurrency/null_lock.rb:9:in 'ActiveSupport::Concurrency::NullLock#synchronize'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:623:in 'ActiveRecord::ConnectionAdapters::TransactionManager#within_new_transaction'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:367:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#within_new_transaction'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:359:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#transaction'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/transactions.rb:234:in 'block in ActiveRecord::Transactions::ClassMethods#transaction'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:412:in 'ActiveRecord::ConnectionAdapters::ConnectionPool#with_connection'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_handling.rb:310:in 'ActiveRecord::ConnectionHandling#with_connection'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/transactions.rb:233:in 'ActiveRecord::Transactions::ClassMethods#transaction'
/var/app/lib/tasks/points.rake:10:in 'block (2 levels) in <main>'
Caused by:
PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_points_on_lonlat_timestamp_user_id" (PG::UniqueViolation)
DETAIL: Key (lonlat, "timestamp", user_id)=(0101000020E6100000FF78AF5A995821C0D099B4A9BA934340, 1744624450, 1) already exists.
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in 'PG::Connection#exec'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in 'ActiveRecord::ConnectionAdapters::PostgreSQL::DatabaseStatements#perform_query'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in 'block (2 levels) in ActiveRecord::ConnectionAdapters::DatabaseStatements#raw_execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:1011:in 'block in ActiveRecord::ConnectionAdapters::AbstractAdapter#with_raw_connection'
/var/app/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/concurrency/null_lock.rb:9:in 'ActiveSupport::Concurrency::NullLock#synchronize'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:983:in 'ActiveRecord::ConnectionAdapters::AbstractAdapter#with_raw_connection'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in 'block in ActiveRecord::ConnectionAdapters::DatabaseStatements#raw_execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/notifications/instrumenter.rb:58:in 'ActiveSupport::Notifications::Instrumenter#instrument'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:1129:in 'ActiveRecord::ConnectionAdapters::AbstractAdapter#log'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#raw_execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#internal_execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:137:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/query_cache.rb:27:in 'ActiveRecord::ConnectionAdapters::AbstractAdapter#execute'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:40:in 'ActiveRecord::ConnectionAdapters::PostgreSQL::DatabaseStatements#execute'
/var/app/lib/tasks/points.rake:14:in 'block (3 levels) in <main>'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:626:in 'block in ActiveRecord::ConnectionAdapters::TransactionManager#within_new_transaction'
/var/app/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/concurrency/null_lock.rb:9:in 'ActiveSupport::Concurrency::NullLock#synchronize'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:623:in 'ActiveRecord::ConnectionAdapters::TransactionManager#within_new_transaction'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:367:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#within_new_transaction'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:359:in 'ActiveRecord::ConnectionAdapters::DatabaseStatements#transaction'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/transactions.rb:234:in 'block in ActiveRecord::Transactions::ClassMethods#transaction'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:412:in 'ActiveRecord::ConnectionAdapters::ConnectionPool#with_connection'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/connection_handling.rb:310:in 'ActiveRecord::ConnectionHandling#with_connection'
/var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/transactions.rb:233:in 'ActiveRecord::Transactions::ClassMethods#transaction'
/var/app/lib/tasks/points.rake:10:in 'block (2 levels) in <main>'
Tasks: TOP => points:migrate_to_lonlat
(See full trace by running task with --trace)
/var/app #
Disclaimer.. I don´t exactly know the datamodel in dawarich, so I don´t know if the procedure below will trash the information in latitude and longitude. Use it at your own risk.
I browsed the data, and I had 137,000 records where longitude and latitude were nil, but raw_data contained the information.
I ran this script to populate latitude and longitude from the raw_data
user = User.find_by(email: '[email protected]') user.tracked_points.where(latitude: nil, longitude: nil).find_each do |point| raw = point.raw_data next unless raw.is_a?(Hash) && raw["lat"].present? && raw["lon"].present? point.latitude = raw["lat"] point.longitude = raw["lon"] point.save(validate: false) endAnd then I could run this to update the lonlat
rake data_cleanup:remove_duplicate_points rake points:migrate_to_lonlatsome helper functions
user = User.find_by(email: '[email protected]') user.tracked_points.where(latitude: nil, longitude: nil).size user.tracked_points.where.not(latitude: nil, longitude: nil).where(lonlat: nil).size
After Running the commands provided here I don't get the nil error when opening the map.
As soon as https://github.com/Freika/dawarich/releases/tag/0.25.5-rc0 is built, you can try using the
freikin/dawarich:rcdocker image to run Dawarich (just replace the image name in your compose file), it contains possible fix for the issue. Therake points:migrate_to_lonlattask now contains additional logic: it takes all points with emptylongitudeandlatitudeattrs and tries to fill them with appropriate data from theraw_datacolumn. The task might take some time and even be killed due to insufficient memory, but running it multiple times until it succeeds is a viable option.Not sure if its related or not but on the RC version I get the following error when running it: Docker in Unraid.
/var/app # bin/rake points:migrate_to_lonlat [dotenv] Loaded .env.development Updating points to use lonlat... WARNING: database "dawarich" has no actual collation version, but a version was recorded D, [2025-04-14T12:03:51.835487 #124] DEBUG -- : Point Load (6.0ms) SELECT "points"."id", "points"."longitude", "points"."latitude", "points"."raw_data" FROM "points" WHERE "points"."longitude" IS NULL AND "points"."latitude" IS NULL ORDER BY "points"."id" ASC LIMIT $1 [["LIMIT", 1000]] D, [2025-04-14T12:03:51.838711 #124] DEBUG -- : ↳ lib/tasks/points.rake:10:in 'block (2 levels) in <main>' rake aborted! NoMethodError: undefined method '[]' for nil (NoMethodError) if point.raw_data['activitySegment']['waypointPath']['waypoints'][0] ^^^^^^^^^^^^^^^^ /var/app/app/services/points/raw_data_lonlat_extractor.rb:21:in 'Points::RawDataLonlatExtractor#extract_lonlat' /var/app/app/services/points/raw_data_lonlat_extractor.rb:9:in 'Points::RawDataLonlatExtractor#call' /var/app/lib/tasks/points.rake:11:in 'block (3 levels) in <main>' /var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:88:in 'Array#each' /var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:88:in 'block in ActiveRecord::Batches#find_each' /var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:172:in 'block in ActiveRecord::Batches#find_in_batches' /var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:461:in 'block in ActiveRecord::Batches#batch_on_unloaded_relation' /var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:434:in 'ActiveRecord::Batches#batch_on_unloaded_relation' /var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:289:in 'ActiveRecord::Batches#in_batches' /var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:171:in 'ActiveRecord::Batches#find_in_batches' /var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.1/lib/active_record/relation/batches.rb:87:in 'ActiveRecord::Batches#find_each' /var/app/lib/tasks/points.rake:10:in 'block (2 levels) in <main>' Tasks: TOP => points:migrate_to_lonlat (See full trace by running task with --trace) /var/app #
@joaoferreira-git let's try with the new version of dawarich:rc image, pull it and try again
Same issue for me, get this screen when trying to log in after updating to 0.25.5 (latest) from yesterday
I have the same issue, even after running the script. I was able to find the '1 minute' having this issue. It is between 2025-04-13 18:32 and 2025-04-13 18:33
Is there a way I can find this record and correct it manually? running bin/rake points:migrate_to_lonlat does not fix this issue. I also tryed my data / points option, but it dies with the same error when I get to the same date
I am on version 0.27.1
Thanks
@schneemaier @michaelcox12 @ethronrion check out this comment and the one below: https://github.com/Freika/dawarich/issues/1327#issuecomment-2945768831
@schneemaier @michaelcox12 @ethronrion check out this comment and the one below: #1327 (comment)
Thank you! Looking at the referenced comment I understand what it is supposed to do. However I have no idea how to execute it, and unfortunatelly currently I don't have time to figure it out.
Send a direct, specific message and I'll try to give you specific instructions. You need access to a database manager that can open the database file. From there, it is a relatively simple procedure.
Send a direct, specific message and I'll try to give you specific instructions. You need access to a database manager that can open the database file. From there, it is a relatively simple procedure.
Hi @TheBig-O, I am not sure what you meant by direct specific message, but here is my setup: dawarich running in docker environment on debian. In this setup how should I access the database manager?
Thanks
@schneemaier @michaelcox12 @ethronrion check out this comment and the one below: #1327 (comment)
Thank you! Looking at the referenced comment I understand what it is supposed to do. However I have no idea how to execute it, and unfortunatelly currently I don't have time to figure it out.
oh, you can run it in the console: https://dawarich.app/docs/FAQ/#how-to-enter-dawarich-console
@schneemaier @michaelcox12 @ethronrion check out this comment and the one below: #1327 (comment)
Thank you! Looking at the referenced comment I understand what it is supposed to do. However I have no idea how to execute it, and unfortunatelly currently I don't have time to figure it out.
oh, you can run it in the console: https://dawarich.app/docs/FAQ/#how-to-enter-dawarich-console
Thank you! I tried it but ended up with error :(
user@server1:~/docker/dawarich# docker exec -it dawarich_app /bin/sh
# bin/rails console
[dotenv] Set DATABASE_PORT
[dotenv] Loaded .env.development
Type 'jobs_help' to see how to connect to the available job servers to manage jobs
Loading development environment (Rails 8.0.2)
[1] pry(main)> Point.where(lonlat: nil).each do |p|
[1] pry(main)> p.update(longitude: p.raw_data['lon'], latitude: p.raw_data['lat'], lonlat: "POINT(#{raw_data['lon']} #{p.raw_data['lat']})")
[1] pry(main)> end
Point Load (10340.8ms) SELECT "points".* FROM "points" WHERE "points"."lonlat" IS NULL
NameError: undefined local variable or method 'raw_data' for main (NameError)
from (pry):2:in 'block in __pry__'
[2] pry(main)>
But when i change the command to
Point.where(lonlat: nil).each do |p|
print p
end
I get exaclty 1 record which do have raw_data field
Point Load (109.1ms) SELECT "points".* FROM "points" WHERE "points"."lonlat" IS NULL
#<Point:0x00007efe855bf8d0>=> [#<Point:0x00007efe855bf8d0
id: 5017388,
battery_status: "charging",
ping: nil,
battery: 45,
tracker_id: "ba",
topic: "owntracks/user/xxxxxx",
altitude: 309,
longitude: nil,
velocity: "0.0",
trigger: "background_event",
bssid: "xxxxxx",
ssid: "XXXXX",
connection: "wifi",
vertical_accuracy: 1,
accuracy: 12,
timestamp: 1744583500,
latitude: nil,
mode: nil,
inrids: nil,
in_regions: nil,
raw_data:
{"m" => 1, "t" => "p", "bs" => 2, "_id" => "5e9d2c60", "acc" => 12, "alt" => xxx, "cog" => 0, "lat" => xx.0030078, "lon" => yy.6264634, "tid" => "ba", "tst" => 1744583500, "vac" => 1, "vel" => 0, "SSID" => "xxxxxxx", "batt" => 45, "conn" => "w", "BSSID" => "xxxxxxxx", "_type" => "location", "point" => {"topic" => "owntracks/user/xxxxxx", "created_at" => 1744585334}, "topic" => "owntracks/user/xxxxxx", "action" => "create", "api_key" => "xxx", "controller" => "api/v1/owntracks/points", "created_at" => 1744585334},
import_id: nil,
city: "x",
country: "x",
created_at: "2025-04-13 19:06:55.155852000 -0400",
updated_at: "2025-04-13 19:07:02.968999000 -0400",
user_id: 1,
geodata:
{"type" => "Feature", "geometry" => {"type" => "Point", "coordinates" => [xx.62643580164813, xx.0029069]}, "properties" => {"city" => "xxx", "type" => "house", "state" => "xxx", "county" => "xxxx", "extent" => [xx.626536, xx.0029969, xx.6263356, xx.0028268], "osm_id" => 245134195, "street" => "xx", "country" => "xx", "osm_key" => "building", "locality" => "xx", "osm_type" => "x", "postcode" => "xx", "osm_value" => "house", "countrycode" => "x", "housenumber" => "x"}},
visit_id: nil,
reverse_geocoded_at: "2025-04-13 19:07:02.779361000 -0400",
course: nil,
course_accuracy: nil,
external_track_id: nil,
lonlat: nil,
country_id: nil>]
I am currently running version 0.27.4
@schneemaier there was a typo in my code, here's the corrected version:
Point.where(lonlat: nil).each do |p|
p.update(longitude: p.raw_data['lon'], latitude: p.raw_data['lat'], lonlat: "POINT(#{p.raw_data['lon']} #{p.raw_data['lat']})")
end
@Freika Thank you, but something is still not working. It finds the point but does not apply the changes, instead I get a rollback notification in the output.
[1] pry(main)> Point.where(lonlat: nil).each do |p|
[1] pry(main)> p.update(longitude: p.raw_data['lon'], latitude: p.raw_data['lat'], lonlat: "POINT(#{p.raw_data['lon']} #{p.raw_data['lat']})")
[1] pry(main)> end
Point Load (137.7ms) SELECT "points".* FROM "points" WHERE "points"."lonlat" IS NULL
TRANSACTION (0.3ms) BEGIN
Point Exists? (1.9ms) SELECT 1 AS one FROM "points" WHERE "points"."lonlat" = $1 AND "points"."id" != $2 AND "points"."timestamp" = $3 AND "points"."user_id" = $4 LIMIT $5 [["lonlat", "[FILTERED]"], ["id", 5017388], ["timestamp", 1744583500], ["user_id", 1], ["LIMIT", 1]]
TRANSACTION (0.4ms) ROLLBACK
=> [#<Point:0x00007f8d535feb68
longitude: [FILTERED],
latitude: [FILTERED],
lonlat: [FILTERED],
user_id: 1,
timestamp: 1744583500,
id: 5017388,
battery_status: "charging",
ping: nil,
battery: 45,
tracker_id: "ba",
topic: "owntracks/user/shiba",
altitude: 309,
velocity: "0.0",
trigger: "background_event",
bssid: "xxxx",
ssid: "xxxx",
connection: "wifi",
vertical_accuracy: 1,
accuracy: 12,
mode: nil,
inrids: nil,
in_regions: nil,
raw_data:
{"m" => 1, "t" => "p", "bs" => 2, "_id" => "5e9d2c60", "acc" => 12, "alt" => 309, "cog" => 0, "lat" => xx.0030078, "lon" => -xx.6264634, "tid" => "ba", "tst" => 1744583500, "vac" => 1, "vel" => 0, "SSID" => "xxxx", "batt" => 45, "conn" => "w", "BSSID" => "xxxx", "_type" => "location", "point" => {"topic" => "owntracks/user/shiba", "created_at" => 1744585334}, "topic" => "owntracks/user/shiba", "action" => "create", "api_key" => "xxxxx", "controller" => "api/v1/owntracks/points", "created_at" => 1744585334},
import_id: nil,
city: "xx",
country: "xx",
created_at: "2025-04-13 19:06:55.155852000 -0400",
updated_at: "2025-04-13 19:07:02.968999000 -0400",
geodata:
{"type" => "Feature", "geometry" => {"type" => "Point", "coordinates" => [-xx.62643580164813, xx.0029069]}, "properties" => {"city" => "xxxx", "type" => "house", "state" => "xxx", "county" => "xxxx", "extent" => [-xx.626536, xx.0029969, -xx.6263356, xx.0028268], "osm_id" => 245134195, "street" => "xxxx", "country" => "xxxxx", "osm_key" => "building", "locality" => "xxxx", "osm_type" => "W", "postcode" => "xxxx", "osm_value" => "house", "countrycode" => "XX", "housenumber" => "XX"}},
visit_id: nil,
reverse_geocoded_at: "2025-04-13 19:07:02.779361000 -0400",
course: nil,
course_accuracy: nil,
external_track_id: nil,
country_id: nil>]
Could it be that you have to pass the numbers as strings? I started using Dawarich just yesterday and had a similar issue, and then noticed in the GPSLogger instructions there are quotes around the numeric values.
What I'm using now on Dawarich 0.27.4 has this structure:
{
"_type": "location",
"t": "u",
"acc": str(data["acc"]),
"alt": str(data["alt"]),
"batt": str(data["batt"]),
"bs": str(data["bs"]),
"lat": str(data["lat"]),
"lon": str(data["lon"]),
"tst": str(data["tst"]),
"vel": str(data["vel"]),
}
(this is python code because I have a middleware forwarding from GPSLogger calls, but here is just copying from the message and converting back to string).
Looks like if the value is invalid you still get an HTTP 200 but then it gives this error processing the message. I don't know Ruby and can't really follow the process, but I assume it's placing the message raw in a queue and processing it later, I can see it in the postgres table.
Could it be that you have to pass the numbers as strings?
I use owntracks to get my location data and that one does not need any special setup. I only have 1 record which is broken, and trying to figure out why the "repair" script is failing.
@schneemaier let's use more explicit version:
Point.where(lonlat: nil).each do |p|
p.update!(longitude: p.raw_data['lon'], latitude: p.raw_data['lat'], lonlat: "POINT(#{p.raw_data['lon']} #{p.raw_data['lat']})")
end
The only change is we're using update! with bang sign, it will explicitly throw an error instead of silently rolling back the transaction
@schneemaier let's use more explicit version:
Point.where(lonlat: nil).each do |p| p.update!(longitude: p.raw_data['lon'], latitude: p.raw_data['lat'], lonlat: "POINT(#{p.raw_data['lon']} #{p.raw_data['lat']})") end The only change is we're using
update!with bang sign, it will explicitly throw an error instead of silently rolling back the transaction
Okay, so the error is suggesting duplicate record?
ActiveRecord::RecordInvalid: Validation failed: Lonlat already has a point at this location and time for this user (ActiveRecord::RecordInvalid)
from /var/app/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.2/lib/active_record/validations.rb:87:in 'ActiveRecord::Validations#raise_validation_error'