How to upload to external database Postgres

[iaguerri]

Hello, i'm trying to create a tileserver in docker (to be deployed in Kubernetes). I need two things:

• The PBF file is available at another url (in a minIO), not geofabrik
• The poly file should not be necessary (if possible)
• The Postgres should be outside of the docker (if possible)

Is there any way to accomplish these? Best regards :)

[iaguerri]

docker run -it -v /c/cc/data-osm-tileserver/datos/spain-latest.osm.pbf:/data/region.osm.pbf -v ./test/data:/data/database overv/openstreetmap-tile-server import

output

+ '[' -z '' ']'
+ echo 'WARNING: No import file at /data/region.osm.pbf, so importing Luxembourg as example...'
WARNING: No import file at /data/region.osm.pbf, so importing Luxembourg as example...
+ DOWNLOAD_PBF=https://download.geofabrik.de/europe/luxembourg-latest.osm.pbf
+ DOWNLOAD_POLY=https://download.geofabrik.de/europe/luxembourg.poly
+ '[' -n https://download.geofabrik.de/europe/luxembourg-latest.osm.pbf ']'
+ echo 'INFO: Download PBF file: https://download.geofabrik.de/europe/luxembourg-latest.osm.pbf'
INFO: Download PBF file: https://download.geofabrik.de/europe/luxembourg-latest.osm.pbf
+ wget https://download.geofabrik.de/europe/luxembourg-latest.osm.pbf -O /data/region.osm.pbf
--2023-10-03 14:15:10--  https://download.geofabrik.de/europe/luxembourg-latest.osm.pbf
Resolving download.geofabrik.de (download.geofabrik.de)... 65.109.50.43, 65.109.48.72
Connecting to download.geofabrik.de (download.geofabrik.de)|65.109.50.43|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 40337746 (38M) [application/octet-stream]
Saving to: ‘/data/region.osm.pbf’

Testing the code above, the Luxembourg file is still downloaded... What I am not doing right?

[Istador]

The check for /data/region.osm.pbf is explicitely looking for a file and not a directory.

That it fails despite having a correct volume mount command could mean that it created a new empty directory instead, because it couldn't find or access the file:

• the path /c/cc/data-osm-tileserver/datos/spain-latest.osm.pbf on the host could be wrong and not exist.
• the docker system isn't configured to be able to volume mount that path (a common issue on Windows, check your docker configuration)

Though I don't think that's the issue, because then it'd have failed downloading and creating the file under the same path as the empty directory.

Because you're using Windows, I'd also suggest to start all absolute paths with two slashes //c/cc/... and //data/region.osm.pbf and //data/database. You might also need to use Windows paths for the host paths instead (e.g. C:\cc\..., C:\\cc\\..., C:/cc/... or similar). Because some 3rd-party terminals try to auto-correct the paths in commands which might lead to invalid paths given to docker. E.g. your file might be mounted, but not at /data/region.osm.pbf but somewhere else in the container.

[iaguerri]

Ok, thanks for the answer! It was the Windows issue.

About the questions related to the external database connection, is there any approach ready yet?

Could it be done like sth as this nominatim does? https://github.com/mediagis/nominatim-docker/issues/245#issuecomment-1072205751 NOMINATIM_DATABASE_DSN="pgsql:dbname=nominatim;hostaddr=192.168.0.3;user=my_user;password=my_pass" \

[galgof-dy]

@iaguerri were you able to work out some way to connect to a postgres instance running /outside/ of the docker container?

[maykazw]

Any info about postgres be outside of the docker ? Its possible ?

[iaguerri]

Hi, yes, it is possible. @galgof-dy , @maykazw

In my case I have to deploy it in k8s, so I used kompose to create the yaml files. The run.sh is a configmap file and I have used this lines to map the db data. The files renderd.conf and project.mml are also added as configmaps. if [ ! -f /data/style/mapnik.xml ]; then cd /data/style/ sed -i "s/dbname: \"gis\"/dbname: \"${OSM_DB}\"/g" project.mml #sed -i "s/dbname: \"${OSM_DB}\"/dbname: \"${OSM_DB}\"/g" project.mml sed -i "/dbname: \"${OSM_DB}\"/a\ user: \"${OSM_USERNAME}\"" project.mml sed -i "/dbname: \"${OSM_DB}\"/a\ password: \"${OSM_PASSWORD}\"" project.mml sed -i "/dbname: \"${OSM_DB}\"/a\ port: \"${OSM_PORT}\"" project.mml sed -i "/dbname: \"${OSM_DB}\"/a\ host: \"${DB_HOST}\"" project.mml carto ${NAME_MML:-project.mml} > prueba-carto.txt carto ${NAME_MML:-project.mml} > mapnik.xml fi

and then in the values.yaml file indicate them as env

envSecret:
  db: "xxxx"
  username: "xxxx"
  password: "xxxxx"
  port: "xxxx"
  host: "xxxxx"

commandContainer:
  enabled: true
  command: "['/bin/bash', '-c', '/opt/run.sh']"

extraEnv:
- name: OSM_DB
  valueFrom:
    secretKeyRef:
      key: OSM_DB
      name: map-bd
- name: OSM_USERNAME
  valueFrom:
    secretKeyRef:
      key: OSM_USERNAME
      name: map-bd
- name: OSM_PASSWORD
  valueFrom:
    secretKeyRef:
      key: OSM_PASSWORD
      name: map-bd
- name: OSM_PORT
  valueFrom:
    secretKeyRef:
      key: OSM_PORT
      name: map-bd
- name: DB_HOST
  valueFrom:
    secretKeyRef:
      key: DB_HOST
      name: map-bd
- name: DOWNLOAD_PBF
  value: https://download.geofabrik.de/europe/spain-latest.osm.pbf
- name: DOWNLOAD_POLY
  value: https://download.geofabrik.de/europe/spain.poly
- name: REPLICATION_URL
  value: https://planet.openstreetmap.org/replication/minute/
- name: UPDATES
  value: enabled
- name: MAX_INTERVAL_SECONDS
  value: "7200"
- name: DB_PORT
  value: "5432"
- name: EXPIRY_MINZOOM
  value: "13"
- name: EXPIRY_TOUCHFROM
  value: "13"
- name: EXPIRY_DELETEFROM
  value: "19"
- name: EXPIRY_MAXZOOM
  value: "20"

I hope this can help!!