osm-parking-processing icon indicating copy to clipboard operation
osm-parking-processing copied to clipboard

[wip] Use schemas for regions

Open gislars opened this issue 1 year ago • 0 comments

currently we import all regions into one schema (import) and all the processing happens in a separate schema (processing). This enables us to continuously update the OSM data and run the processing independently from the import.

When importing a new region we only want to provide a new pbf and region boundary. But there is a chance (a very little one) that the updating goes wrong when we mix data with different timestamps and try to update them. One solution would be to only import all regions at once and make sure that all have the same timestamp. One could do this with the configuration provided in extracts/*.

This approach has some disadvantages:

  • create and import all extracts
  • downtime while updating
  • with each new region a whole import is needed

One idea to solve this is to use separate db schemas for each region. Advantages:

  • regions can be updated, inserted, deleted independently from each other
  • no mixup of timestamps, each region can start with an individual timestamp (not too old!) and osm2pgsql will update it
  • for each region a individual settings could be used
    • update interval
    • projection

Disadvantages:

  • for each region a separate update job is running
  • multiple import and middle schemas use more storage space

This issue is about to keep track of this topic, collect experiences and progress.

gislars avatar Mar 18 '23 20:03 gislars