wiktorn
Initialization hangs at: "OSM file has no replication headers. Looking for newest OSM object."
Hi there, i am trying to initialize the overpass api with these environment vars:
"Env": [
"OVERPASS_RULES_LOAD=10",
"OVERPASS_META=yes",
"OVERPASS_MODE=init",
"OVERPASS_PLANET_URL=https://download.geofabrik.de/europe-latest.osm.bz2",
"OVERPASS_DIFF_URL=https://download.openstreetmap.fr/replication/europe/minute/",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NGINX_VERSION=1.27.1",
"NJS_VERSION=0.8.5",
"NJS_RELEASE=1~bookworm",
"PKG_RELEASE=1~bookworm",
"DYNPKG_RELEASE=2~bookworm",
"OTEL_VERSION=0.1.0",
"OVERPASS_USE_AREAS=true",
"OVERPASS_ALLOW_DUPLICATE_QUERIES=no"
],
the machine i am running it on is a 32GB Ram / 16Core machine hostet at hetzner.de. i am currently wondering if it somehow crashed or not, since i have the same log message for more than a day:
2025-07-06 01:52:34 DEBUG: OSM file has no replication headers. Looking for newest OSM object.
which generally makes sense since i have pulled it from geofabrik. if i look at htop i can see that there are processes working on the update:
but there is currently nothing in the diffs folder:
(base) root@machine:/big/docker/overpass_europe# ls
changes.log cookie.jar db/ diffs/ planet.osm.bz2
(base) root@machine:/big/docker/overpass_europe# ls diffs/
(base) root@machine:/big/docker/overpass_europe# ls db/
node_frequent_tags.bin nodes.bin.idx node_tags_global.bin relation_frequent_tags.bin relation_roles.bin.idx relations_meta.bin relation_tags_local.bin.idx user_indices.bin.idx ways.bin ways_meta.bin.idx
node_frequent_tags.bin.idx nodes.map node_tags_global.bin.idx relation_frequent_tags.bin.idx relations.bin relations_meta.bin.idx rules way_frequent_tags.bin ways.bin.idx way_tags_global.bin
node_keys.bin nodes.map.idx node_tags_local.bin relation_keys.bin relations.bin.idx relation_tags_global.bin user_data.bin way_frequent_tags.bin.idx ways.map way_tags_global.bin.idx
node_keys.bin.idx nodes_meta.bin node_tags_local.bin.idx relation_keys.bin.idx relations.map relation_tags_global.bin.idx user_data.bin.idx way_keys.bin ways.map.idx way_tags_local.bin
nodes.bin nodes_meta.bin.idx osm_base_version relation_roles.bin relations.map.idx relation_tags_local.bin user_indices.bin way_keys.bin.idx ways_meta.bin way_tags_local.bin.idx
(base) root@machine:/big/docker/overpass_europe#
these are the file sizes:
du -sch *
4.0K changes.log
4.0K cookie.jar
80G db
4.0K diffs
49G planet.osm.bz2
129G total
is there some missconfiguration going on, or do i just need to wait another day or two?
currently the docker container reports it is running since 4 days (this is what it took to process europe) and unhealthy:
4e6c2f89d791 wiktorn/overpass-api "/docker-entrypoint.…" 4 days ago Up 4 days (unhealthy) 0.0.0.0:12345->80/tcp, [::]:12345->80/tcp overpass_europe
to me it seems it is taking way too long, but i am not sure if it might be ok. should i restart the container? or just leave it?
Thanks a lot for help!
I've reached a similar point after 44 hour of init. About 5.25 hours later, the latest timestamp was found, and within less than a second, the corresponding sequence ID was found.
I was under the impression that in init mode, the .osm.bz2 file is scanned for the latest timestamp by the
osmium fileinfo -e -g data.timestamp.last /db/planet.osm.bz2 command within
https://github.com/wiktorn/Overpass-API/blob/5396bf55c10c380bb03c9bcdb192814e2c1de3ea/docker-entrypoint.sh#L89
As far as I can tell, that scan took about 3.5 hours.
@wiktorn,
Do you think it would be useful to capture that last timestamp and pass it to update_overpass.sh
https://github.com/wiktorn/Overpass-API/blob/5396bf55c10c380bb03c9bcdb192814e2c1de3ea/docker-entrypoint.sh#L94
which would add it as a --start-date flag to
https://github.com/wiktorn/Overpass-API/blob/5396bf55c10c380bb03c9bcdb192814e2c1de3ea/bin/update_overpass.sh#L51
Some timestamps from docker compose logs --timestamps overpass, FYI.
Downloading the planet data:
overpass-1 | 2025-07-16T19:00:17.129481630Z No database directory. Initializing
overpass-1 | 2025-07-16T19:00:17.136533058Z % Total % Received % Xferd Average Speed Time Time Time Current
overpass-1 | 2025-07-16T19:00:17.136564387Z Dload Upload Total Spent Left Speed
100 341 100 341 0 0 8872 0 --:--:-- --:--:-- --:--:-- 8973
100 384 100 384 0 0 7566 0 --:--:-- --:--:-- --:--:-- 7566
100 152G 100 152G 0 0 34.6M 0 1:14:56 1:14:56 --:--:-- 39.1M1:14:50 1:12:35 0:02:15 30.5M
overpass-1 | 2025-07-16T20:15:13.379337171Z Running preprocessing command:
End of the first scan for the latest timestamp :
overpass-1 | 2025-07-17T01:51:32.036555163Z Reading XML file ... elapsed node 1. Compute current ... ready. Flushing to database ....... done.
The database creation completion:
overpass-1 | 2025-07-18T15:05:43.002259241Z Update complete.
overpass-1 | 2025-07-18T15:05:43.181244239Z Database created. Now updating it.
overpass-1 | 2025-07-18T15:05:43.185340497Z Updating
overpass-1 | 2025-07-18T15:05:49.200599256Z 2025-07-18 15:05:49 DEBUG: OSM file has no replication headers. Looking for newest OSM object.
End of the second scan for the latest timestamp:
overpass-1 | 2025-07-18T20:21:36.106824278Z 2025-07-18 20:21:36 INFO: Using replication server at https://planet.openstreetmap.org/replication/minute/
overpass-1 | 2025-07-18T20:21:36.107181203Z 2025-07-18 20:21:36 DEBUG: Looking up sequence ID for timestamp 2025-07-06 23:59:59+00:00
Identification of the sequence ID
overpass-1 | 2025-07-18T20:21:36.375772665Z 2025-07-18 20:21:36 DEBUG: Starting download at ID 6673397 (max 100 MB)
