dawarich icon indicating copy to clipboard operation
dawarich copied to clipboard
verified publisher icon Freika

OwnTracks data not being processed by Sidekiq, database relations issue?

Open DanielB1990 opened this issue 10 months ago • 1 comments

OS & Hardware Running Unraid 6.12.13 ( Slackware 15.0 x86_64 )

Dell PowerEdge R730 CPU: Intel Xeon E5-2630L v3 (32) @ 2.900GHz GPU: Matrox Electronics Systems Ltd. G200eR2 RAM: 62 GiB

Version Dawarich ( app & sidekiq ), 0.24.1 Docker version 24.0.9, build 2936816 | - redis:latest | - postgres:16-bullseye

FROM tensorchord/pgvecto-rs-binary:pg16-v0.2.0-amd64 as pgvectorsbinary

FROM postgres:16-bullseye

RUN localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8
ENV TZ=Europe/Amsterdam

# Avoid stuck build due to user prompt
ARG DEBIAN_FRONTEND=noninteractive

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN dpkg-reconfigure -f noninteractive tzdata

# Debian 11 <= /etc/apt/sources.list
RUN if [ -f "/etc/apt/sources.list" ]; then sed -i 's/deb.debian.org/ftp.nl.debian.org/g' /etc/apt/sources.list; fi
# Debian 12 > /etc/apt/sources.list.d/debian.sources
RUN if [ -f "/etc/apt/sources.list.d/debian.sources" ]; then sed -i 's/deb.debian.org/ftp.nl.debian.org/g' /etc/apt/sources.list.d/debian.sources; fi

RUN apt-get -qq update
RUN apt-get -qq update --fix-missing
RUN apt-get -qq install -y --no-install-recommends ca-certificates postgresql-16-postgis-3 postgresql-16-postgis-3-scripts
RUN apt-get -qq autoclean -y
RUN apt-get -qq autoremove -y
RUN rm -rf /var/lib/apt/lists/*

RUN mkdir -p /docker-entrypoint-initdb.d

COPY --from=pgvectorsbinary /pgvecto-rs-binary-release.deb /tmp/vectors.deb
RUN apt-get -qq install -y /tmp/vectors.deb && rm -f /tmp/vectors.deb
RUN cp -a /usr/lib/postgresql/16/lib/vectors.so /usr/share/postgresql/16/extension/

COPY ./load-extensions.sh /docker-entrypoint-initdb.d/

CMD ["postgres"]

load-extensions.sh

#!/bin/sh
psql -a -b -e -v ON_ERROR_STOP=1 --username=postgres -no-password -c 'CREATE EXTENSION IF NOT EXISTS pg_trgm;'
psql -a -b -e -v ON_ERROR_STOP=1 --username=postgres -no-password -c 'CREATE EXTENSION IF NOT EXISTS postgis;'
psql -a -b -e -v ON_ERROR_STOP=1 --username=postgres -no-password -c 'CREATE EXTENSION IF NOT EXISTS postgis_topology;'
psql -a -b -e -v ON_ERROR_STOP=1 --username=postgres -no-password -c 'CREATE EXTENSION IF NOT EXISTS postgis_tiger_geocoder;'
psql -a -b -e -v ON_ERROR_STOP=1 --username=postgres -no-password -c 'CREATE EXTENSION IF NOT EXISTS postgis_raster;'
psql -a -b -e -v ON_ERROR_STOP=1 --username=postgres -no-password -c 'CREATE EXTENSION IF NOT EXISTS postgis_sfcgal;'
psql -a -b -e -v ON_ERROR_STOP=1 --username=postgres -no-password -c 'CREATE EXTENSION IF NOT EXISTS address_standardizer;'
psql -a -b -e -v ON_ERROR_STOP=1 --username=postgres -no-password -c 'CREATE EXTENSION IF NOT EXISTS fuzzystrmatch;'
psql -a -b -e -v ON_ERROR_STOP=1 --username=postgres -no-password -c 'CREATE EXTENSION IF NOT EXISTS postgis_tiger_geocoder;'
psql -a -b -e -v ON_ERROR_STOP=1 --username=postgres -no-password -c 'ALTER SYSTEM SET shared_preload_libraries = "vectors.so"'

| -- Extensions:

bash /docker-entrypoint-initdb.d/load-extensions.sh
CREATE EXTENSION IF NOT EXISTS pg_trgm;
NOTICE:  extension "pg_trgm" already exists, skipping
CREATE EXTENSION IF NOT EXISTS postgis;
NOTICE:  extension "postgis" already exists, skipping
CREATE EXTENSION IF NOT EXISTS postgis_topology;
NOTICE:  extension "postgis_topology" already exists, skipping
CREATE EXTENSION IF NOT EXISTS postgis_tiger_geocoder;
NOTICE:  extension "postgis_tiger_geocoder" already exists, skipping
CREATE EXTENSION IF NOT EXISTS postgis_raster;
NOTICE:  extension "postgis_raster" already exists, skipping
CREATE EXTENSION IF NOT EXISTS postgis_sfcgal;
NOTICE:  extension "postgis_sfcgal" already exists, skipping
CREATE EXTENSION IF NOT EXISTS address_standardizer;
NOTICE:  extension "address_standardizer" already exists, skipping
CREATE EXTENSION IF NOT EXISTS fuzzystrmatch;
NOTICE:  extension "fuzzystrmatch" already exists, skipping
CREATE EXTENSION IF NOT EXISTS postgis_tiger_geocoder;
NOTICE:  extension "postgis_tiger_geocoder" already exists, skipping
ALTER SYSTEM SET shared_preload_libraries = "vectors.so"

Describe the bug I use OwnTracks ( 2.5.3 via Google Play Store, Android ) , and no points show up when pushing them manually ( for testing ) I noticed that Sidekiq is having issues processing the points, making relations to:

  • users
  • points

To Reproduce

  1. Start up a fresh 0.24.1 Dawarich ( app & sidekiq ) with redis:latest and the mentioned dockerfile for postgres.
  2. Login with demo
  3. Change user credentials
  4. Regenerate api key
  5. Link to OwnTracks
  6. docker logs -f -t $(docker ps | grep -i 'Dawarich-Sidekiq' | awk '{print $1}')
  7. Push a location and see the error

Expected behavior That my location is tracked as expected by the usage of Dawarich

Screenshots Not applicable

Logs dawarich_app

2025-02-21T10:44:55.608460463Z 11:44:55 web.1  | D, [2025-02-21T11:44:55.607962 #21] DEBUG -- :   User Load (63.6ms)  SELECT "users".* FROM "users" WHERE "users"."api_key" = $1 LIMIT $2  [["api_key", "[FILTERED]"], ["LIMIT", 1]]
2025-02-21T10:44:55.609276949Z 11:44:55 web.1  | D, [2025-02-21T11:44:55.608953 #21] DEBUG -- :   ↳ app/controllers/api_controller.rb:16:in 'ApiController#current_api_user'
2025-02-21T10:44:55.616016838Z 11:44:55 web.1  | I, [2025-02-21T11:44:55.615624 #21]  INFO -- : Enqueued Owntracks::PointCreatingJob (Job ID: 8a5bfe8c-4e9a-4030-a26f-38a8cf9aa1d4) to Sidekiq(default) with arguments: #<ActionController::Parameters {"_type" => "location", "BSSID" => "XX:XX:XX:XX:XX:XX", "SSID" => "WifiSSIDName", "_id" => "464b36e7", "acc" => 11, "alt" => 58, "batt" => 78, "bs" => 1, "cog" => 0, "conn" => "w", "created_at" => 1740134695, "lat" => 52.masked, "lon" => 6.masked, "m" => 0, "t" => "u", "tid" => "1s", "topic" => "owntracks/user/mymobilephone", "tst" => 1740134693, "vac" => 1, "vel" => 0, "api_key" => "b1108017fd4ec928masked", "controller" => "api/v1/owntracks/points", "action" => "create", "point" => #<ActionController::Parameters {"topic" => "owntracks/user/mymobilephone", "created_at" => 1740134695} permitted: true>} permitted: true>, 1
2025-02-21T10:44:55.617026236Z 11:44:55 web.1  | I, [2025-02-21T11:44:55.616718 #21]  INFO -- : ↳ app/controllers/api/v1/owntracks/points_controller.rb:5:in 'Api::V1::Owntracks::PointsController#create'
2025-02-21T10:44:55.617581150Z 11:44:55 web.1  | I, [2025-02-21T11:44:55.617275 #21]  INFO -- : {"method":"POST","path":"/api/v1/owntracks/points","format":"html","controller":"Api::V1::Owntracks::PointsController","action":"create","status":200,"allocations":4854,"duration":111.44,"view":0.23,"db":77.4}
2025-02-21T10:44:55.771792150Z 11:44:55 web.1  | D, [2025-02-21T11:44:55.771262 #21] DEBUG -- :   User Load (1.3ms)  SELECT "users".* FROM "users" WHERE "users"."api_key" = $1 LIMIT $2  [["api_key", "[FILTERED]"], ["LIMIT", 1]]
2025-02-21T10:44:55.773541704Z 11:44:55 web.1  | D, [2025-02-21T11:44:55.773119 #21] DEBUG -- :   ↳ app/controllers/api_controller.rb:16:in 'ApiController#current_api_user'
2025-02-21T10:44:55.778058346Z 11:44:55 web.1  | I, [2025-02-21T11:44:55.776997 #21]  INFO -- : Enqueued Owntracks::PointCreatingJob (Job ID: 02fe02d4-0f68-4637-b5e9-734adc5025be) to Sidekiq(default) with arguments: #<ActionController::Parameters {"_type" => "status", "_id" => "8141e698", "android" => #<ActionController::Parameters {"hib" => 1, "bo" => 0, "loc" => 0, "ps" => 0, "wifi" => 1} permitted: true>, "topic" => "owntracks/user/mymobilephone/status", "api_key" => "b1108017fd4ec928masked", "controller" => "api/v1/owntracks/points", "action" => "create", "point" => #<ActionController::Parameters {"topic" => "owntracks/user/mymobilephone/status"} permitted: true>} permitted: true>, 1
2025-02-21T10:44:55.778526975Z 11:44:55 web.1  | I, [2025-02-21T11:44:55.778190 #21]  INFO -- : ↳ app/controllers/api/v1/owntracks/points_controller.rb:5:in 'Api::V1::Owntracks::PointsController#create'
2025-02-21T10:44:55.782031885Z 11:44:55 web.1  | I, [2025-02-21T11:44:55.781708 #21]  INFO -- : {"method":"POST","path":"/api/v1/owntracks/points","format":"html","controller":"Api::V1::Owntracks::PointsController","action":"create","status":200,"allocations":1298,"duration":32.15,"view":3.23,"db":1.33}

dawarich_sidekiq, see attached sidekiq-logs.txt file

Additional context Not applicable, I did try bin/rails db:migrate and rake data:migrate to be sure all migrations are done and that seems to be the case:

docker exec -it $(docker ps | grep 'Dawarich-app' | awk '{print $1}') /bin/sh

/var/app # bin/rails db:migrate
[dotenv] Loaded .env.development
D, [2025-02-21T12:03:41.331046 #383] DEBUG -- :    (1.1ms)  SELECT pg_try_advisory_lock(1300285682869233375)
D, [2025-02-21T12:03:41.356180 #383] DEBUG -- :   ActiveRecord::SchemaMigration Load (3.2ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
D, [2025-02-21T12:03:41.406214 #383] DEBUG -- :   ActiveRecord::InternalMetadata Load (2.4ms)  SELECT * FROM "ar_internal_metadata" WHERE "ar_internal_metadata"."key" = $1 ORDER BY "ar_internal_metadata"."key" ASC LIMIT 1  [[nil, "environment"]]
D, [2025-02-21T12:03:41.410698 #383] DEBUG -- :    (0.6ms)  SELECT pg_advisory_unlock(1300285682869233375)
D, [2025-02-21T12:03:41.420904 #383] DEBUG -- :   ActiveRecord::SchemaMigration Load (0.5ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC

/var/app # rake data:migrate
[dotenv] Loaded .env.development
D, [2025-02-21T12:03:50.046681 #393] DEBUG -- :    (2.1ms)  SELECT pg_try_advisory_lock(1300285682869233375)
D, [2025-02-21T12:03:50.071405 #393] DEBUG -- :   DataMigrate::DataSchemaMigration Load (3.4ms)  SELECT "data_migrations"."version" FROM "data_migrations" ORDER BY "data_migrations"."version" ASC
D, [2025-02-21T12:03:50.095340 #393] DEBUG -- :   ActiveRecord::InternalMetadata Load (4.7ms)  SELECT * FROM "ar_internal_metadata" WHERE "ar_internal_metadata"."key" = $1 ORDER BY "ar_internal_metadata"."key" ASC LIMIT 1  [[nil, "environment"]]
D, [2025-02-21T12:03:50.128651 #393] DEBUG -- :    (0.7ms)  SELECT pg_advisory_unlock(1300285682869233375)
D, [2025-02-21T12:03:50.130598 #393] DEBUG -- :   DataMigrate::DataSchemaMigration Load (0.4ms)  SELECT "data_migrations"."version" FROM "data_migrations" ORDER BY "data_migrations"."version" ASC

DanielB1990 avatar Feb 21 '25 11:02 DanielB1990

I've gone this way, since I already had redis ( nothing special configured ) and PostgreSQL running ( added extensions to debian base image ) , but taking the default compose file from https://dawarich.app/docs/intro#setup-your-dawarich-instance works as expected/ so probably something wrong with the PostgreSQL docker file I'm using. Will have to debug myself.

DanielB1990 avatar Feb 24 '25 14:02 DanielB1990

Please update to the most recent version and check if your issue persists. Coordinate with updating guides to not miss anything important

Freika avatar Jun 05 '25 18:06 Freika

Closing due to no activity

Freika avatar Aug 27 '25 10:08 Freika