tilemaker
tilemaker copied to clipboard
systemed
Planet tile generation -- results and experience
Hi, this is less a specific issue and more a recount of my experience generating tiles for planet.osm.pbf in the past few days. I have been keenly following the developments of this tool, which are all very exciting. Kudos especially to @systemed and @kleunen who have clearly made tremendous advances recently.
For background, I haven't been involved in OSM or mapping generally -- but I am working on a web service for which mapping will play a role -- particularly the display and plotting of items on world map tiles. I have not generated any OSM map tiles before this recent experience, so this is all new to me and I've likely made some mistakes along the way.
For all my work I've used Google Cloud Platform -- mostly VMs and persistent disks in this case. I know @kleunen's contributions have brought the needed RAM usage way down but in this case I still used a rather large instance to complete the task (though without the inclusion of the mmap functionality I believe it would still not have succeeded even on this machine). Specifically I used an n2-highmem-80 instance (80 vCPUs, 640GB memory) -- with a 4TB SSD persistent disk attached.
Now from what I can tell the recent optimizations are extremely exciting because they can vastly lower the cost of generating these tiles. This appears to be true in my case. I want to caution that I don't have specific timings for anything on this run (though I did grab some system screenshots at a few spots). However I was able to split the run into three parts, each of which completed in under 24 hours (importance of which is explained below):
- Read in planet.osm.pbf + generate mvt files for zoom 1-13
- Read in planet.osm.pbf + generate mvt files for a portion of zoom 14
- Read in planet.osm.pbf + generate mvt files for the rest of zoom 14
What can be important about finishing each (resumable) step in under 24 hours is that it means I can use a preemptible VM and thereby save a bit over 70% of the hourly cost over a normal long-running instance. (Google shuts off a preemptible instance at the end of every 24 hours if not earlier -- fortunately not earlier during this run.)
I worked off the b8c7a78f86ba15528bf3ef691ab22968785cd15e commit and modified to compile with LuaJIT (which I think increased the speed of the read-in process by close to 2x). I also increased the initial mmap size to 1TB -- original definition is here: https://github.com/systemed/tilemaker/blob/b8c7a78f86ba15528bf3ef691ab22968785cd15e/include/osm_store.h#L272 so that it wouldn't have to resize during the run. Also I was able to accomplish the split between steps 2 and 3 by modifying tilemaker.cpp on the last run to exclude a hardcoded list of tiles that had already been generated previously.
I generated according to provided config-openmaptiles.json / process-openmaptiles.lua except at all zoom levels and excluding house numbers. I just generated individual MVT tile files directly instead of a .mbtiles file.
The read-in of planet.osm.pbf took I believe over half the run-time on each of the three steps -- and as @kleunen mentioned there would likely be a lot of benefits to parallelizing it. I could see that during this portion only 1 vCPU (out of 80 available) was being used -- and only a similar small fraction of the possible disk IO. But of course RAM and extra mmap disk usage was appropriately high.
(The top command as shown above above usually had the 1 vCPU pegged at 100 during the read-in but nearing the end it dropped a bit.)
Of course during the parallelized tile generation stage the system resources were more fully utilized:
Now one issue I had which may or might not be a mistake on my part somewhere is that the most zoomed out tiles (zooms 1, 2, and 3) were empty. I still need to investigate, again it might be user error on that.
At the lower zooms I do have though it is apparent that https://github.com/systemed/tilemaker/issues/203 is a valid suggestion -- as currently things are pretty empty. Additionally https://github.com/systemed/tilemaker/issues/196 will be pretty crucial (assuming that's the reason I don't have borders).
However the higher zooms look great:
I was very happy to be able to complete this tile generation run! I will definitely need to redo it with fixes for the above issues, but for now I'll be able to use what I have for developing the mapping portion of my project and later (likely in a couple of months perhaps) regenerate before making the functionality publicly available.
In terms of cost, given the three days I had the preemtible machine running + disk usage (in reality a bit under three days because the steps didn't take the full 24 hours) , I believe it will cost about $150 for this tile generation. So of course if the process could be brought under 24 hours (perhaps through more parallelization or other changes) that would lead to a cost of more like $50. Needless to say that's orders of magnitude less than what a third party would charge for these tiles (as far as I've seen). And given that I had to read the PBF three times under this methodology, I think the true, non-interrupted runtime is already under 48 hours on this machine type.
Anyway I hope this writeup is somewhat useful. With respect to the concrete suggestions that I have:
- Ensure zoom levels 1-3 get generated properly (but again, this might be my fault, I'm really not sure yet)
- Follow through with https://github.com/systemed/tilemaker/issues/196 and https://github.com/systemed/tilemaker/issues/203
- As @kleunen mentioned, pbf reading could benefit from parallelization
- Have a built-in means of resuming a run if it gets cut short (maybe tricky for the read-in phase, but definitely doable for the tile-generation phase perhaps at the cost of another read-in of the pbf)
This is truly great work by the contributors, I just want to thank you again. I hope I can help in whatever way, though I may not be able to contribute code directly nor run the process again except for at significant milestones (as even though it's now inexpensive compared to alternatives, it's still a significant cost to run regularly).
Thank you for sharing this information, it is very insightful. Regarding the z1-z3 zoom levels, I am generating Europe country by country now. But I am seeing the same thing as you are seeing, no details at z1-z3. I think this is due to the layer configuration and not a mistake on your part. If you look at the config-openmaptiles.json you can also see that the minimum zoom level is 4. I think maptiler and the configs have simply not been optimized for maps of this scale before.
Regarding the parallel reading, i think that will make a big difference. But it may take some time to implement, making sure no threading issues occur and the application does not crash in the process will take some time. But for now there is another approach I was thinking about that may make better use of the resources you have.
I made a Makefile which generated the mbtiles for specific regions. You can use the same the regions Europe/Asia/Africa, etc .. You can download the region abstracts from Geofabrik or you can actually use their polygons to generate the extract for the different regions .
You can then generate the tiles for the different regions in parallel. You can do that by using the make -j8 command for example. This way the loading is parallelized. After all the regions are generated, you can join them again using tile-join.
Just to point out, the issue that you are both experiencing with low levels, is partly due to the missing globallandcover layer that I've talked about right here https://github.com/systemed/tilemaker/issues/203 precisely yesterday.
It's quite clear that the OpenMapTiles style isn't matching our needs / desires in those lower zoom levels. Although, the good news is that it shouldn't be such a hard task to achieve it, due to the small amount of tiles in those levels. I'm working on it and hope to be able to provide more information in the above issue. Although, I'd be more than happy to accept collaborators in this matter as well.
@carlos-mg89 Also, i am trying out your suggestion now to generating the different Europe countries and using tile-join to join them into one Europe map. I am almost finished in generating the individual country tiles. But you have not replyed my question yet, if there will be any artifacts / double roads / polygons or any issues from joining the different mbtiles. Did you see any issues when using the tile-join when joining the different countries in one Europe map ?
@carlos-mg89 But if I look at the maptiler basic style, at the higher zoom levels, I only see the borders and only for countries where it is available. Country borders and regional borders and some names of cities. Not saying the globallandcover layer it not useful, but I think with only the borders, a basic style for higher zoom layers would be possible and acceptable for my application.
@dmrotar This is great - really good to hear your experience and that it worked for you. A couple of thoughts:
modified to compile with LuaJIT (which I think increased the speed of the read-in process by close to 2x)
If you have any hints for this (or even a PR!) that'd be great. Currently it'll build with LuaJIT fine on macOS but I've not been able to get it to do so on Ubuntu.
Have a built-in means of resuming a run if it gets cut short
Couple of possibilities here. When outputting .mbtiles, we could fairly easily have a --resume option that ran a SELECT COUNT(*) FROM tiles WHERE zoom_level=? AND tile_column=? AND tile_row=? LIMIT 1 before generating a tile, and skipped it if one was found. There'd be some overhead but not too much.
Alternatively, we could write a .tilemaker_progress file every 1000 tiles or so, and then just pick up from that when we restart.
Ensure zoom levels 1-3 get generated properly
I'll have a go at tackling #196, at which point it should in theory be possible to generate a fairly rudimentary set of small-scale tiles. From then on we can look at refining it with landcover etc.
Separately it might be interesting to work up a small-scale style based on Natural Earth rather than OSM, which would enable producing a world map for low zooms really quickly.
Once you have generated the osm_store.dat, you can actually recover the complete OSMStore from this. You actually would not have to reload the pbf again, you can just continue with the tile generation. But this assumes the osm_store.dat is completely valid, so not that the application has crashed while loading the pbf. So maybe after loading the pbf, some header needs to be written the store is valid. And also the tile information and output objects and attributes, they need to be part of the osm_store. This would allow to continue the tile generation at a later stage without reloading the pbf.
But maybe this is a bit too advanced. Probably it is easier to generate the regions seperately and join the tiles.
@kleunen I think the tile-join tool from tippecanoe takes care of the proper merging of lines, polygons, etc. I haven't experienced any issue with this process, and I've used it for more than a year now. You can do a check yourself, by mergin 2 MBTiles and then go to the borders. It's the first I did, and I have to say that the result was great. I didn't appreciate any issue.
@carlos-mg89 Ok thank you. I tried merging now, but tileserver-php gives me the error: JSON metadata is not valid. Please check configuration of your server. Not sure what that is
I use tessella as MBTiles server and I'm really happy with it. The configuration is very simple.
You just need to install tessella, and then you have to start it (by default at port 4000). I don't know anything about tileserver-php.
This command will run in the background:
nohup tessella mbtiles://./europe_base.mbtiles &>/dev/null &
@systemed I just checked my changes regarding LuaJIT. I used the libluajit-5.1-dev package, and ran on Debian 10.8. For whatever reason, I needed to comment out these lines in kaguya.hpp to get it to compile: https://github.com/systemed/tilemaker/blob/b8c7a78f86ba15528bf3ef691ab22968785cd15e/include/kaguya.hpp#L726-L730
@kleunen and @carlos-mg89, thanks for your tips regarding usage of tile-join -- I will definitely look into that for the next run. I was initially concerned about discontinuities or missing features at the seams but sounds like you can confirm that there aren't those kinds of issues, which is great!
@kleunen I've not used tile-join but I'd guess that the .mbtiles it's creating don't contain the additional metadata that some clients need - either that, or it's malformed in some way.
You could load one of your original .mbtiles files into sqlite3, copy the relevant row from the metadata table (i.e. SELECT value FROM metadata WHERE name=json), and insert it into the newly merged file.
Yes. I think it is maybe malformed. There is a json row. Which seems really big.
That's very possible, I remember having to deal with this, a long time ago. Right now I don't remember the details, but I do remember having the metadata table being removed. Using tessella you won't have issues with it though.
I thought maybe i could extract the files to a directory with flat files. But no luck so far. Maybe later this week i will have a look.
With the tile-join, i have it working now. There was indeed something wrong with the metadata. The json value was not correct, I copied this value from a tilemaker generated mbtiles file. MBtiles writes information about the layers in this row. After this, it was loading, but not displaying. Maybe it was just centered incorrectly. I removed the 'center' row from metadata and changed the 'type' to 'basemap' (this was overlay before). So:
json -> tilemaker generated json layers
center -> remove
type -> 'basemap'
I also had a look at making maptilers loading parallel, but it is really not simple. Just adding mutexes to the different data structures (osm store, tile data, output objects) is simply not going to cut it. I tried this and it simply does not scale up. This is because loading the map actually involves a lot of heap allocations and building this data structure in memory. The problem however is that the heap is a shared resource. Only a single thread can perform heap allocation at the same time. So this end up all the threads just waiting for each other to get access to the heap memory. In the end, the approach was slower than just using a single thread.
So somehow you need to "chop up" the work for tilemaker to do. This can be either in the dimension of lat/lon or zoom (layers). I guess people are using these approaches already. The advantage of seperating by layer is that you do not have to perform extract/join operation on the map. osmium extract is pretty clever, but there is always a possibility of some "errors". Seperating by layer is from a quality of map point of view, the best in my opinion.
One thing I have tried out is to "save" the state of tilemaker after loading the pbf. This effectively creates a mmap index of the loaded pbf file. A subsequent run of tilemaker can then skip the pbf loading and just mmap this file, and continue to the generation of the geometries and generating the tiles.
I noticed writing the actual pbf entries into a mmap file does take some time, but this approach is possible. This way, if you generate multiple layers (in parallel or in sequence), it is possible to save some time from loading the pbf file. Also, when running in parallel, it saves memory. Because the nodes/ways/relations store of tilemaker can be shared between processes (all tilemaker processes mmap the same index file).
If someone is interested in evaluating this, I can create a WIP PR from my work to try out.
What I also see is that when zoomed out, you sometimes see these grid patterns occuring:
when I zoom, they are gone:
and in tileserver-php i see this:
Hm. That might be a result of the filtering by area that we do in the Lua profile - any small areas won't be written at lower zooms, so if a seemingly big forest is actually made up of lots of smaller areas, then they'll disappear. But it is a bit suspicious that a few of them are seemingly aligned to tile boundaries.
You can actually see the same effect on maptiler basic. Although, maybe not as severe
Another good reason to create some landcover polygons for low zoom levels :)
I also tried using LuaJIT (libluajit-5.1-dev) with the Ubuntu20.04 Docker image. "lua_tonumberx" error could be resolved with the following fix:
kaguya.hpp
diff --git a/include/kaguya.hpp b/include/kaguya.hpp
index ef54bdc..e0c1611 100644
--- a/include/kaguya.hpp
+++ b/include/kaguya.hpp
@@ -5124,7 +5124,7 @@ namespace kaguya
static get_type get(lua_State* l, int index)
{
int isnum=0;
- get_type num = static_cast<T>(lua_tonumberx(l,index,&isnum));
+ get_type num = static_cast<T>(kaguya::lua_tonumberx(l,index,&isnum));
if (!isnum) {
throw LuaTypeMismatch();
}
Makefile
- LUA_LIBS := -lluajit
+ LUA_LIBS := -lluajit-5.1
I may be wrong but.
Looks good, force the compile to use kaguya::lua_tonumberx
I also have to enable -fPIE to build
I also h
I also tried using LuaJIT (libluajit-5.1-dev) with the Ubuntu20.04 Docker image. "lua_tonumberx" error could be resolved with the following fix:
kaguya.hpp
diff --git a/include/kaguya.hpp b/include/kaguya.hpp index ef54bdc..e0c1611 100644 --- a/include/kaguya.hpp +++ b/include/kaguya.hpp @@ -5124,7 +5124,7 @@ namespace kaguya static get_type get(lua_State* l, int index) { int isnum=0; - get_type num = static_cast<T>(lua_tonumberx(l,index,&isnum)); + get_type num = static_cast<T>(kaguya::lua_tonumberx(l,index,&isnum)); if (!isnum) { throw LuaTypeMismatch(); }Makefile
- LUA_LIBS := -lluajit + LUA_LIBS := -lluajit-5.1I may be wrong but.
Have a look here: #206
@kleunen recently this project https://github.com/woltapp/parallelpbf came into my knowledge, where they ease parallel reading / writing of OSM PBF files. Maybe it's of any use for you ;)
I've been working on other related stuff (generation of proper contours and hill shades), so couldn't do much.
What I can confirm is that the OpenMapTiles generator has improved drastically since last year when I last used it. I needed to get Europe generated, including the country & states boundaries, along with other details I'm missing in the tiles generated with tilemaker (sadly). And it took me between 4 and 5 days to read and generate the whole Europe MBTiles, with 32 GB of RAM and taking about 350 GB of hard drive. What impressed me the most is that it didn't use the whole RAM at all.
Yes. What i think is that performance is not the biggest issue anymore. I think indeed the missing level of detail is the biggest problem now. I do not know how difficult that is to fix. Possibly by tuning the config and process file, much better results can be achieved? Especially at lower zoom levels? I honestly have to little experience to judge about that.
Reading the pbf parallel is definitly possible. It would not be too difficult. But I measured this, and actually the decompression and loading, it is only a small part. Most cpu is used for allocating the memory for the linestrings/relations I think.
Once you have generated the osm_store.dat, you can actually recover the complete OSMStore from this. You actually would not have to reload the pbf again, you can just continue with the tile generation.
Is it possible for tilemaker to complete after writing the .dat and then resume based on it? That'd split the work and allow for parallelizing of the first part but not the second.
Yes, it is possible with the --index
--index generate an index file from the specified input
file
When you restart with --index, and the osm.pbf has a matching index file, the index will be used instead of reading the .osm.pbf. But after this, still, the geometries are generated by executing the lua files. You can create multiple config files with different minzoom/maxzoom settings and generate the different zoom levels in parallel.
I am not sure how well generating in parallel will work with an mbtiles files. This is a sqlite3 file and I think tilemaker completely locks this files, so your best performance will be generating into a directory.
Another approach will be by using osmium to split the planet into regions, I will explain in the message after this.
I think another and more efficient approach would be to use Osmium Extract to extract 4 regions:
(-180,-90,0,0) (0,-90,180,0) (0,0,180,90) (-180,0,0,90)
If you create a config file for osmium extract, you can actually extract these 4 regions in one run. Be sure to include the --set-bounds command line option to osmium, because tilemaker needs the bounds from the osm.pbf file.
I got these files from the planet: 57G Mar 6 05:35 planet-latest.osm.pbf 2.3G Apr 6 15:05 planet_1.renum.osm.pbf 4.3G Apr 6 15:30 planet_2.renum.osm.pbf 34G Apr 6 11:10 planet_3.osm.pbf 17G Apr 6 14:31 planet_4.osm.pbf
Planet 1 region will start the tile generation first (which uses all CPU), and then other regions will follow.
These regions are actually the four tiles from zoom level 1 (if I am not mistaken). So what you can do is generate into a directory and skip zoom level 1. Or you generate 4 mbtiles files and use tippecanoe tile-join.
Be sure to give every tilemake it's own --store file. Because if multiple are writing to the same file, it will crash.
Hello,
Here are my results for planet generation using "old" hardware:
Hardware specs:
- Dell R620 running XCP-NG, working VM has 200Gb RAM, 8 CPU (E5-2609)
- Input & output files stored on a NFS Server (1Gbps interface)
- Local disk to store temp file on a Samsung NVME 2Tb disk (store used 500Gb file)
Here is the output:
Stored 6869005895 nodes, 761835990 ways, 4732804 relations
Shape points: 0, lines: 742524, polygons: 67901
Generated points: 164927374, lines: 414908973, polygons: 507887983
Zoom level 14, writing tile 358057903 of 358057903
Filled the tileset with good things at /data/map/output/planet.mbtiles
real 5946m29,650s
user 2m3,945s
sys 3m49,487s
This project is awesome ! Thanks !!
