dawarich icon indicating copy to clipboard operation
dawarich copied to clipboard

Reverse geocoding stuck at zero

Open zzyzx-dc opened this issue 3 months ago • 10 comments

Perhaps I'm just doing something incorrectly, but I have a selfhosted photon instance and have downloaded my region. Photon is in the same docker compose file and I am able to curl json information from host photon about locations when I enter into the dawarich_app and dawarich_sidekiq containers.

I am not sure where to proceed if the containers can see each other, and the photon container can provide location data. Perhaps I screwed up my compose file somewhere:

compose.yaml
networks:
  dawarich:
  lsio:
    external: true

services:
  dawarich_redis:
    image: redis:7.4-alpine
    container_name: dawarich_redis
    command: redis-server
    networks:
      - dawarich
    volumes:
      - dawarich_shared:/data
    restart: always
    healthcheck:
      test: [ "CMD", "redis-cli", "--raw", "incr", "ping" ]
      interval: 10s
      retries: 5
      start_period: 30s
      timeout: 10s

  dawarich_db:
    image: postgis/postgis:17-3.5-alpine
    shm_size: 1G
    container_name: dawarich_db
    volumes:
      - dawarich_db_data:/var/lib/postgresql/data
      - dawarich_shared:/var/shared
      # - ./postgresql.conf:/etc/postgresql/postgresql.conf # Optional, uncomment if you want to use a custom config
    networks:
      - dawarich
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: dawarich_development
    restart: always
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres -d dawarich_development" ]
      interval: 10s
      retries: 5
      start_period: 30s
      timeout: 10s
    # command: postgres -c config_file=/etc/postgresql/postgresql.conf # Use custom config, uncomment if you want to use a custom config

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    volumes:
      - dawarich_public:/var/app/public
      - dawarich_watched:/var/app/tmp/imports/watched
      - dawarich_storage:/var/app/storage
      - dawarich_db_data:/dawarich_db_data
    networks:
      - dawarich
      - lsio
    ports:
      - 3001:3000
      # - 9394:9394 # Prometheus exporter, uncomment if needed
    stdin_open: true
    tty: true
    entrypoint: web-entrypoint.sh
    command: ['bin/rails', 'server', '-p', '3000', '-b', '::']
    restart: on-failure
    environment:
      PHOTON_API_HOST: photon:2322  # use your Photon host + port 
      PHOTON_API_USE_HTTPS: "false"
      RAILS_ENV: ${RAILS_ENV}
      REDIS_URL: redis://dawarich_redis:6379
      DATABASE_HOST: dawarich_db
      DATABASE_USERNAME: postgres
      DATABASE_PASSWORD: ${DATABASE_PASSWORD}
      DATABASE_NAME: dawarich_development
      MIN_MINUTES_SPENT_IN_CITY: 60
      APPLICATION_HOSTS: "localhost,dawarich.URL.com"
      TIME_ZONE: Europe/London
      APPLICATION_PROTOCOL: http
      PROMETHEUS_EXPORTER_ENABLED: "false"
      PROMETHEUS_EXPORTER_HOST: 0.0.0.0
      PROMETHEUS_EXPORTER_PORT: 9394
      SELF_HOSTED: "true"
      STORE_GEODATA: "true"
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"
    healthcheck:
      test: [ "CMD-SHELL", "wget -qO - http://127.0.0.1:3000/api/v1/health | grep -q '\"status\"\\s*:\\s*\"ok\"'" ]
      interval: 10s
      retries: 30
      start_period: 30s
      timeout: 10s
    depends_on:
      dawarich_db:
        condition: service_healthy
        restart: true
      dawarich_redis:
        condition: service_healthy
        restart: true
    deploy:
      resources:
        limits:
          cpus: '0.50'    # Limit CPU usage to 50% of one core
          memory: '4G'    # Limit memory usage to 4GB

  dawarich_sidekiq:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    volumes:
      - dawarich_public:/var/app/public
      - dawarich_watched:/var/app/tmp/imports/watched
      - dawarich_storage:/var/app/storage
    networks:
      - dawarich
    stdin_open: true
    tty: true
    entrypoint: sidekiq-entrypoint.sh
    command: ['sidekiq']
    restart: on-failure
    environment:
      PHOTON_API_HOST: photon:2322  # use your Photon host + port 
      PHOTON_API_USE_HTTPS: "false"
      RAILS_ENV: ${RAILS_ENV}
      REDIS_URL: redis://dawarich_redis:6379
      DATABASE_HOST: dawarich_db
      DATABASE_USERNAME: postgres
      DATABASE_PASSWORD: ${DATABASE_PASSWORD}
      DATABASE_NAME: dawarich_development
      APPLICATION_HOSTS: localhost
      BACKGROUND_PROCESSING_CONCURRENCY: 10
      APPLICATION_PROTOCOL: http
      PROMETHEUS_EXPORTER_ENABLED: "false"
      PROMETHEUS_EXPORTER_HOST: dawarich_app
      PROMETHEUS_EXPORTER_PORT: 9394
      SELF_HOSTED: "true"
      STORE_GEODATA: "true"
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"
    healthcheck:
      test: [ "CMD-SHELL", "pgrep -f sidekiq" ]
      interval: 10s
      retries: 30
      start_period: 30s
      timeout: 10s
    depends_on:
      dawarich_db:
        condition: service_healthy
        restart: true
      dawarich_redis:
        condition: service_healthy
        restart: true
      dawarich_app:
        condition: service_healthy
        restart: true

  photon:
    image: rtuszik/photon-docker:latest
    environment:
      - UPDATE_STRATEGY=PARALLEL
      - UPDATE_INTERVAL=720h # Check for updates every 30 days
      # - REGION=andorra # Optional: specific region (continent, country, or planet)
      - REGION=us
      # - APPRISE_URLS=pover://user@token  # Optional: notifications
    volumes:
          - photon_data:/photon/data
    restart: unless-stopped
    ports:
            - "2322:2322"
    networks:
      - dawarich

volumes:
  photon_data:
  dawarich_db_data:
  dawarich_shared:
  dawarich_public:
  dawarich_watched:
  dawarich_storage:

zzyzx-dc avatar Sep 21 '25 00:09 zzyzx-dc

edit: Resolved - see comment below.

alangrainger avatar Sep 22 '25 11:09 alangrainger

@zzyzx-dc I had the same issue. Just now I saw this comment https://github.com/Freika/dawarich/issues/1781#issuecomment-3314974209 and added PHOTON_* to the sidekiq container, and all my points are now processing.

alangrainger avatar Sep 22 '25 11:09 alangrainger

@zzyzx-dc will that fix the issue for you?

Freika avatar Sep 22 '25 20:09 Freika

@zzyzx-dc will that fix the issue for you?

I believe I already have this incorporated in my compose file. Please see my compose file in the original post and double-check if I have it correctly.

I have

 PHOTON_API_HOST: photon:2322  # use your Photon host + port 
  PHOTON_API_USE_HTTPS: "false"

Added to both the app and the sidekiq containers

zzyzx-dc avatar Sep 26 '25 04:09 zzyzx-dc

An update: I have 1,546 points of 64,000 geocoded over the last four days. It's moving, but moving quite slow.

This can't possibly be normal or expected operation, can it?

zzyzx-dc avatar Sep 26 '25 04:09 zzyzx-dc

A week later:

3,484 points geocoded

64,146 points without data

zzyzx-dc avatar Oct 03 '25 02:10 zzyzx-dc

@zzyzx-dc in the console, run Geocoder.config and make sure you don't have komoot's photon in there

Freika avatar Oct 07 '25 18:10 Freika

Thanks. I believe this is pointed at my selfhosted photon instance...not sure! "Lookup: photon" and "host photon:2322" look like they're pointing at my 'photon' host.

$ docker exec -it dawarich_app /bin/sh
# bin/rails console

[dotenv] Set DATABASE_PORT
[dotenv] Loaded .env.development
Loading development environment (Rails 8.0.2.1)

[1] pry(main)> Geocoder.config
=> {timeout: 5,
 lookup: :photon,
 ip_lookup: :ipinfo_io,
 language: :en,
 http_headers: {"User-Agent" => "Dawarich 0.32.0 (https://dawarich.app)"},
 use_https: false,
 http_proxy: nil,
 https_proxy: nil,
 api_key: nil,
 basic_auth: {},
 logger: :kernel,
 kernel_logger_level: 2,
 always_raise: :all,
 units: :km,
 distances: :linear,
 cache: #<Redis client v5.4.0 for redis://dawarich_redis:6379>,
 cache_prefix: nil,
 cache_options: {expiration: 1 day},
 debug_mode: true,
 photon: {host: "photon:2322"}}

Although I realize my compose file names the service 'photon' but the container 'dawarich_photon' - is that an issue?

zzyzx-dc avatar Oct 08 '25 01:10 zzyzx-dc

I also think that there's no geocoding in the queue. This is my sidekiq dashboard:

21,214 Processed 6,307 Failed 0 Busy 0 Enqueued 0 Retries 0 Scheduled 0 Dead

And I ran this:

$ docker exec -it dawarich_sidekiq bin/rails c [dotenv] Set DATABASE_PORT [dotenv] Loaded .env.development Loading development environment (Rails 8.0.2.1) [1] pry(main)> Sidekiq::Queue.all.map { |q| [q.name, q.size] } INFO 2025-10-08T01:07:10.005Z pid=772 tid=298: Sidekiq 8.0.4 connecting to Redis with options {size: 10, pool_name: "internal", url: "redis://dawarich_redis:6379/1"} => 
[["cache", 0], 
["default", 0], 
["imports", 0], 
["places", 0], 
["points", 0], 
["reverse_geocoding", 0], 
["stats", 0], 
["tracks", 0], 
["trips", 0], 
["visit_suggesting", 0]] 
[2] pry(main)> exit

CPU and memory usage for the container is very low, so that's not an issue.

I am up to 4,000+ points geocoding, so progress is obviously being made....why would the queue be empty though?

zzyzx-dc avatar Oct 08 '25 01:10 zzyzx-dc

In the UI, go to Settings -> Background Jobs -> Continue reverse geocoding, and then check the queue. It should contain points that were not reverse geocoded and basically a new attempt of revese geocoding should be taken for them

Freika avatar Oct 11 '25 12:10 Freika