planetiler
planetiler copied to clipboard
onthegomap
[BUG] Error occurred during stage lake_centerlines
Describe the bug I am getting this error when I run the command from the readme.
To Reproduce Steps to reproduce the behavior:
- Run:
docker run -e JAVA_TOOL_OPTIONS="-Xmx1g" -v "$(pwd)/data":/data ghcr.io/onthegomap/planetiler:latest --download --area=monaco \
--water-polygons-url=https://github.com/onthegomap/planetiler/raw/main/planetiler-core/src/test/resources/water-polygons-split-3857.zip \
--natural-earth-url=https://github.com/onthegomap/planetiler/raw/main/planetiler-core/src/test/resources/natural_earth_vector.sqlite.zip
- See the error:
0:00:00 INF - Using merge sort feature map, chunk size=357mb max workers=8
Exception in thread "main" com.onthegomap.planetiler.Planetiler$PlanetilerException: Error occurred during stage lake_centerlines
at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:845)
at org.openmaptiles.OpenMapTilesMain.run(OpenMapTilesMain.java:58)
at org.openmaptiles.OpenMapTilesMain.main(OpenMapTilesMain.java:18)
at com.onthegomap.planetiler.Main.main(Main.java:102)
Caused by: java.io.UncheckedIOException: java.util.zip.ZipException: zip END header not found
at com.onthegomap.planetiler.util.FileUtils.walkPathWithPattern(FileUtils.java:102)
at com.onthegomap.planetiler.util.FileUtils.walkPathWithPattern(FileUtils.java:113)
at com.onthegomap.planetiler.Planetiler.lambda$addShapefileSource$3(Planetiler.java:365)
at com.onthegomap.planetiler.Planetiler.lambda$ifSourceUsed$13(Planetiler.java:971)
at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:843)
... 3 more
Caused by: java.util.zip.ZipException: zip END header not found
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.findEND(ZipFileSystem.java:1338)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.initCEN(ZipFileSystem.java:1559)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.<init>(ZipFileSystem.java:179)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.getZipFileSystem(ZipFileSystemProvider.java:125)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:120)
at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:528)
at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:475)
at com.onthegomap.planetiler.util.FileUtils.walkPathWithPattern(FileUtils.java:78)
... 7 more
Expected behavior It doesn't give an error.
Environment (please complete the following information):
- Hardware: Apple M1 Pro
- OS: 14.6.1
- Java version and distribution: using docker
- Maven version: using docker
Additional context
Also happening with the full dataset when running:
docker run -e JAVA_TOOL_OPTIONS="-Xmx1g" -v "$(pwd)/data":/data ghcr.io/onthegomap/planetiler:latest --download --area=monaco
Is this still happening? And what about if you delete the data directory so it re-downloads them? I tried those commands and they are working fine for me.
What was the solution to this? I get the same error. Have tried deleting data folder but after download it happens again. I download the lake_centerline file on another windows pc and it looks fine. same zip file size as on ubuntu instance. I am using the latest planetiler container.
I built a new Ubuntu server from scratch to try to resolve this, and still the same issue. Tried the exact same command line as above that fails also. No one else experiencing this?
Update: I went into the sources folder, renamed the lake_centerline file, did a wget of the file directly, and now it appears to be running properly, even though both old and new files are exactly the same size. I had been running this process monthly for over a year without issue... I'd still love to know what was causing this if anyone else has seen it.
Hi @jatech1! I got the exact same issue yesterday.
The end of the log says:
1:55:16 INF [download] - Setting map bounds from input: Envelope(-180.0,-90.0,180.0,90.0)
1:55:16 DEB [download] - argument: archive_name=OpenMapTiles ('name' attribute for tileset metadata)
1:55:16 DEB [download] - argument: archive_description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org ('description' attribute for tileset metadata)
1:55:16 DEB [download] - argument: archive_attribution=<a href="https://www.openmaptiles.org/" target="_blank">© OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">© OpenStreetMap contributors</a> ('attribution' attribute for tileset metadata)
1:55:16 DEB [download] - argument: archive_version=3.15.0 ('version' attribute for tileset metadata)
1:55:16 DEB [download] - argument: archive_type=baselayer ('type' attribute for tileset metadata)
1:55:16 DEB [download] - argument: archive_format=pbf ('format' attribute for tileset metadata)
1:55:16 DEB [download] - argument: compact=true (mbtiles: reduce the DB size by separating and deduping the tile data)
1:55:16 DEB [download] - argument: no_index=false (mbtiles: skip adding index to sqlite DB)
1:55:16 DEB [download] - argument: vacuum_analyze=false (mbtiles: vacuum analyze sqlite DB after writing)
1:55:16 INF [download] - Using merge sort feature map, chunk size=2000mb max workers=16
Exception in thread "main" com.onthegomap.planetiler.Planetiler$PlanetilerException: Error occurred during stage lake_centerlines
at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:880)
at org.openmaptiles.OpenMapTilesMain.run(OpenMapTilesMain.java:58)
at org.openmaptiles.OpenMapTilesMain.main(OpenMapTilesMain.java:18)
at com.onthegomap.planetiler.Main.main(Main.java:102)
Caused by: java.io.UncheckedIOException: java.util.zip.ZipException: zip END header not found
at com.onthegomap.planetiler.util.FileUtils.walkPathWithPattern(FileUtils.java:102)
at com.onthegomap.planetiler.util.FileUtils.walkPathWithPattern(FileUtils.java:113)
at com.onthegomap.planetiler.Planetiler.lambda$addShapefileSource$3(Planetiler.java:366)
at com.onthegomap.planetiler.Planetiler.lambda$ifSourceUsed$14(Planetiler.java:1006)
at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:878)
... 3 more
Caused by: java.util.zip.ZipException: zip END header not found
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.findEND(ZipFileSystem.java:1338)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.initCEN(ZipFileSystem.java:1559)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.<init>(ZipFileSystem.java:179)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.getZipFileSystem(ZipFileSystemProvider.java:125)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:120)
at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:528)
at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:475)
at com.onthegomap.planetiler.util.FileUtils.walkPathWithPattern(FileUtils.java:78)
After downloading the file manually I actually noticed a difference:
xxx@xxx /xxx/xxx/sources # diff lake_centerline.shp.zip lake_centerline.shp.zip.old
Binary files lake_centerline.shp.zip and lake_centerline.shp.zip.old differ
So I think the download from maptiler may have been broken at this time.
Indeed manually downloading with curl yields a difference with that downloaded by planetiler. Even after lake_centerline is removed, the new file downloaded by planetiler differs from curl. The curl version does work.
@irisjae Did your download come from maptiler or did you use planetiler repo that overrides the maptiler sources with their canonical URLs?
I was using https://github.com/openmaptiles/planetiler-openmaptiles and the URL indicated in the run was https://dev.maptiler.download/geodata/omt/lake_centerline.shp.zip, so I guess it comes from maptiler.
The good news is, that this is not fatal: Once you get the lake_centerline.shp.zip say with curl it is kept and you can churn out tilesets.
The bad news is, that I so far failed to replicate this, hence I do not know what is the root cause and how to fix that. Sorry for that.
Then also sort of bad news (from reproducibility point of view): CI jobs for planetiler (until #1238 ) and planetiler-openmaptiles (still) use the maptiler.download URL without this problem.
I'll keep looking.
Related issue which I got first time today (I also bumped earlier to lake centerlines bug, but got it fixed with guidance from @msbarry to replace source for the zip-file). I'm using planetiler via customised planetiler-openmaptiles profile for OSM Europe from Geofabrik, running as a automated job in a container / k3s pod (thus "manual download of files" fix not feasible, but can naturally implement new processing steps). My pipeline has been running mostly without issues now for a year or so.
0:01:17 INF [natural_earth] - Starting...
Exception in thread "main" com.onthegomap.planetiler.Planetiler$PlanetilerException: Error occurred during stage natural_earth
at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:877)
at org.openmaptiles.OpenMapTilesMain.run(OpenMapTilesMain.java:59)
at org.openmaptiles.OpenMapTilesMain.main(OpenMapTilesMain.java:18)
Caused by: java.lang.IllegalArgumentException: java.util.zip.ZipException: zip END header not found
at com.onthegomap.planetiler.reader.NaturalEarthReader.<init>(NaturalEarthReader.java:64)
at com.onthegomap.planetiler.reader.NaturalEarthReader.lambda$process$0(NaturalEarthReader.java:87)
at com.onthegomap.planetiler.reader.SourceFeatureProcessor.getFeatureCount(SourceFeatureProcessor.java:159)
at com.onthegomap.planetiler.reader.SourceFeatureProcessor.processFiles(SourceFeatureProcessor.java:78)
at com.onthegomap.planetiler.reader.SourceFeatureProcessor.processFiles(SourceFeatureProcessor.java:65)
at com.onthegomap.planetiler.reader.NaturalEarthReader.process(NaturalEarthReader.java:84)
at com.onthegomap.planetiler.Planetiler.lambda$addNaturalEarthSource$6(Planetiler.java:510)
at com.onthegomap.planetiler.Planetiler.lambda$ifSourceUsed$14(Planetiler.java:1003)
at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:875)
... 2 more
Caused by: java.util.zip.ZipException: zip END header not found
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.findEND(ZipFileSystem.java:1320)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.initCEN(ZipFileSystem.java:1541)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.<init>(ZipFileSystem.java:179)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.getZipFileSystem(ZipFileSystemProvider.java:125)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:120)
at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:528)
at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:475)
at com.onthegomap.planetiler.reader.NaturalEarthReader.open(NaturalEarthReader.java:96)
at com.onthegomap.planetiler.reader.NaturalEarthReader.<init>(NaturalEarthReader.java:62)
... 10 more
Failed to generate after 5 attempts
Which version of planetiler does your fork pull in? I added some extra debugging in 0.9.0
@msbarry 0.8.4 (back from 0.8.5-snapshot that I used prior to lake centerlines fetch issue). Can I migrate to 0.9.0 without updating planetiler-openmaptiles (my fork is 9 commits behind i.e. based on Mar 31 2025 version).
Yes that should be fine. Also, in order to fix this issue, I'll need a way to reliably reproduce it so any clues you can find would be helpful!
@msbarry I did not have time on the weekend to build & test new image, however here are the results from last 4 daily runs: first 2 failures, followed by 2 successful runs (no changes in the execution env). Understand the challenge in reliably reproducing this...
Btw, on "our end" (e.g. MapTiler's https://dev.maptiler.download/... ): We still do not know what is the root cause but colleagues plan to do some tweaks to the server. I'll keep watching.
Having the same issue with natural_earth. When just downloading it on my mac, it works, when i download the file that the server got, I am unable to unzip it on my mac. Upgrading to v0.9.0 to see if i got some better logs now 👍
These are my error logs. I just fixed it by do a curl to fetch the zip before I start.
0:00:01 INF [download:lake_centerlines] - Finished downloading https://dev.maptiler.download/geodata/omt/lake_centerline.shp.zip to data/sources/lake_centerline.shp.zip
0:00:04 ERR [download:natural_earth] - Error downloading https://dev.maptiler.download/geodata/omt/natural_earth_vector.sqlite.zip to data/sources/natural_earth_vector.sqlite.zip
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Unexpected EOF at 14187 expecting 433384440
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.onthegomap.planetiler.util.Downloader.httpDownload(Downloader.java:305)
at com.onthegomap.planetiler.util.Downloader.lambda$downloadIfNecessary$1(Downloader.java:211)
at com.onthegomap.planetiler.worker.RunnableThatThrows.runAndWrapException(RunnableThatThrows.java:16)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
at java.base/java.util.concurrent.ThreadPerTaskExecutor$TaskRunner.run(ThreadPerTaskExecutor.java:314)
at java.base/java.lang.VirtualThread.run(VirtualThread.java:311)
Caused by: java.lang.IllegalStateException: Unexpected EOF at 14187 expecting 433384440
at com.onthegomap.planetiler.util.Downloader.lambda$httpDownload$3(Downloader.java:300)
... 4 more
0:00:09 INF [download:water_polygons] - Finished downloading https://osmdata.openstreetmap.de/download/water-polygons-split-3857.zip to data/sources/water-polygons-split-3857.zip
Exception in thread "main" com.onthegomap.planetiler.util.Exceptions$FatalPlanetilerException: java.util.concurrent.ExecutionException: com.onthegomap.planetiler.util.Exceptions$FatalPlanetilerException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Unexpected EOF at 14187 expecting 433384440
at com.onthegomap.planetiler.util.Exceptions.throwFatalException(Exceptions.java:29)
at com.onthegomap.planetiler.stats.ProgressLoggers.await(ProgressLoggers.java:367)
at com.onthegomap.planetiler.stats.ProgressLoggers.awaitAndLog(ProgressLoggers.java:352)
at com.onthegomap.planetiler.util.Downloader.run(Downloader.java:179)
at com.onthegomap.planetiler.Planetiler.download(Planetiler.java:1039)
at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:838)
at org.openmaptiles.OpenMapTilesMain.run(OpenMapTilesMain.java:58)
at org.openmaptiles.OpenMapTilesMain.main(OpenMapTilesMain.java:18)
Caused by: java.util.concurrent.ExecutionException: com.onthegomap.planetiler.util.Exceptions$FatalPlanetilerException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Unexpected EOF at 14187 expecting 433384440
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
at com.onthegomap.planetiler.stats.ProgressLoggers.await(ProgressLoggers.java:361)
... 6 more
Caused by: com.onthegomap.planetiler.util.Exceptions$FatalPlanetilerException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Unexpected EOF at 14187 expecting 433384440
at com.onthegomap.planetiler.util.Exceptions.throwFatalException(Exceptions.java:29)
at com.onthegomap.planetiler.worker.RunnableThatThrows.runAndWrapException(RunnableThatThrows.java:18)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
at java.base/java.util.concurrent.ThreadPerTaskExecutor$TaskRunner.run(ThreadPerTaskExecutor.java:314)
at java.base/java.lang.VirtualThread.run(VirtualThread.java:311)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Unexpected EOF at 14187 expecting 433384440
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.onthegomap.planetiler.util.Downloader.httpDownload(Downloader.java:305)
at com.onthegomap.planetiler.util.Downloader.lambda$downloadIfNecessary$1(Downloader.java:211)
at com.onthegomap.planetiler.worker.RunnableThatThrows.runAndWrapException(RunnableThatThrows.java:16)
... 3 more
Caused by: java.lang.IllegalStateException: Unexpected EOF at 14187 expecting 433384440
at com.onthegomap.planetiler.util.Downloader.lambda$httpDownload$3(Downloader.java:300)
... 4 more
@msbarry....first 2 failures, followed by 2 successful runs (no changes in the execution env). Understand the challenge in reliably reproducing this...
This report 4 days ago has been followed by more successful runs, not a single failure. And I did not yet update to 0.9.0, will wait until the next failure. Neither I do any pre-fetching of the data, just rely on the planetiler downloading the data.
This is the exact same thing I was seeing before... the downloader gets the first ~14kb of the file but then the server responds with -1 (EOF). Only ever from dev.maptiler.com URLs. I think that's behind cloudflare, maybe it is imposing some sort of rate limit or throttling?
Another thing we could try is to implement retries in this case, so when it reaches a premature EOF, open another range request starting where it left off. Possibly up to the limit set by HTTP retries cli argument...
@msbarry After many successful runs (once a day) I got a failure again this morning. Same error log (still on 0.8.4) as above in my May 16 comment.
Yup, looks same here:
https://github.com/openmaptiles/planetiler-openmaptiles/actions/runs/15295688777/job/43024138131?pr=235
E.g. "Download data" step of "Build / Run" CI check failed:
Error: Exception in thread "main" com.onthegomap.planetiler.util.Exceptions$FatalPlanetilerException: java.util.concurrent.ExecutionException: com.onthegomap.planetiler.util.Exceptions$FatalPlanetilerException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Unexpected EOF at 14189 expecting 80961538
I think there are 2 things we could do on the planetiler side to try better handle flaky servers like dev.maptiler.com, either retry the entire download when a part fails due to a premature EOF, or try opening a new range request starting at where the server left off when this happens.
@msbarry @phanecak-maptiler After many successful runs (once a day) I have now gotten 2 consecutive failures on this same error. Is more debugging info still needed (from upgrading to 0.9.0)? I guess no other fix is available at this time except to implement pre-fetch solution as done e.g. by @MichielMortier?
No clue at our end, so yes, you can use pre-fetch as workaround. Sorry for the inconvenience.
In the mean time I've started digging in the download code to work on what @msbarry mentioned:
I think there are 2 things we could do on the planetiler side to try better handle flaky servers like dev.maptiler.com ...
I have the same problem with https://github.com/acalcutt/osm-lakelines/releases/download/v12/lake_centerline.shp.zip
redownloading does not seem to fix it
Edit: same problem when downloading the MapTiler hosted one
OK I'm merging https://github.com/onthegomap/planetiler/pull/1270 which adds automated retries in this case. It should be the default you get from docker latest once this action finishes: https://github.com/onthegomap/planetiler/actions/runs/15760034060/job/44424284419