operations
operations copied to clipboard
openstreetmap
Host vector tiles generated by tilemaker
The OWG is planning to host vector tiles generated by tilemaker as an experiment. The goal is to get something others can work with to see where more work needs to be done
The intended architecture is
- weekly downloads of the planet file and generation with tilemaker (est. 2-3 day runtime) to create a mbtiles artifact
- running a mbtiles server, perhaps https://github.com/systemed/tilemaker/blob/master/server/server.rb#L30 behind apache
- potentially offering the raw mbtiles for download
- reusing the IP rate-limiting code for 429s
- using fastly as a CDN
We have some resources we can potentially reuse for this, but the immediate priority is scaling up tile.osm.org with the new dublin and umea servers.
Items that need addressing are
- [x] How much space does a mbtiles file for the planet need? @systemed
- [x] How much RAM does tilemaker need for the planet? @systemed
- [ ] Is there a way to reuse parts of another Fastly config to avoid duplication, particularly of useragent blocks
I know you indicate this to be an experiment but still - may i ask what the strategic vision is behind this or in other words: Where do you intend this to go if the experiment should be successful (by whatever definition of successful)?
I mean is it supposed to lead to the OSMF providing a vector tile service for other parties (and if yes: how does it intend to position that relative to commercial services in that domain)? Or is it supposed to be a non-monetary support program/grant for tilemaker development (and if yes: on what basis was it decided to choose tilemaker to receive that support and not other software in the domain of map rendering/data preprocessing for map rendering)?
As you know i have been saying for quite some time that the OSMF needs to strategically invest in development of technologies suitable for scalable high quality cartography based on OSM data and usable for cooperative community projects so in principle i would see favorably any steps in that direction. But i would like to understand how this step fits into such a strategy or if not what other strategic goals it is meant to serve.
The immediate goal is to support the development of vector tiles for openstreetmap.org.
Trying to understand what you mean by that - so this is a feasibility study to see if tilemaker can be practically managed operationally to generate world wide vector tiles (with no requirements what that entails specified so far, presumably within the current capabilities of tilemaker) with a weekly update cycle and then recruit some volunteer style developers to develop a map style (or several - presumably using Maputnik) to produce maps targeting yet to be specified needs on openstreetmap.org.
I see a number of issues with that (in particular in the long term regarding the strategic needs for community map styles providing meaningful constructive feedback for mappers). But this is evidently not a very suitable venue to discuss such matters (although i would very much suggest you to have such a discussion with a broader reach - not many people follow this tracker and few would be inclined to comment here).
Independent of the specifics discussed here i would suggest to clearly distinguish between operational work (stuff that serves concrete and well specified immediate needs) and strategic endeavors and programs. When you state the goal to support the development of vector tiles for openstreetmap.org it is not clear if that is an operational goal (in which case it seems a bit underspecified - see above) or strategic endeavor (in which case a more long term vision what strategic goals this should serve would be highly advisable IMO).
What I said on the operations call yesterday (and this is my personal view) is that I hoped that providing them would break the impasse that seems to exist where people can't work on a style because there's no tiles but don't work on tiles because there's no style.
My hope would be that people will start developing a vector style we can use and that there can then be a feedback cycle between that and any necessary changes to the tile schema until we have something we feel is deployable.
At that point we can worry about what stack to use in production and how best to generate and serve vector tiles for that.
Understood. Just keep in mind that when the OSMF rolls out something like that there will be a message of endorsement (of the tools used, their capabilities and their limitations and the standards they are built upon) and a message of strategic direction (in what direction community cartography in the OSM community should develop) perceived even if no such message is intended.
(edit: not relevant to this ticket)
~~This is exciting! I'm new to this effort, but I'm wondering it the intent here is to offer an experimental vector layer visible on OSM.org? I predict needing some messaging to manage expectations around it not being an alternative or successor "basemap" to OSM Carto (even if that is a long term goal, it would depend on heavy development of a cartographic product like Tom described)~~
~~What would be immediately useful is if the vector layer provided both:~~
~~1) a language switcher to change the name: tag rendered by by the browser~~
~~2) integration with OSM.org User Preferences language settings to default to those language tags~~
~~Even if the vector layer displays only place= points+relations and no other geometries, it would be immediately useful as a tile-based web visualization of multilingual data already in OSM; OSM Carto only shows name, and iD only works at very high zoom levels. Even better would be if the rendered name tags were clickable links to a /node/12345/edit page.~~
Please can we keep this ticket on topic and not go off on crazy tangents.
This ticket is just about getting an experimental vector tile source deployed.
Writing a vector style is a matter for other people to be discussed in other places.
Deploying that style to openstreetmap.org is a matter for operations and/or the web site developers once it exists, which is a logn way off.
Once we have the tiles we will probably put some sort of minimal style up on a separate domain as a way of visualising them in the short term, with the hope that somebody will then start to develop a full scale style that we can then put in it's place until it's ready to consider a production deployment.
We are all well aware of the additional opportunities that vector tiles present, which is why we are trying to encourage people to work in that direction, but this is not the place to discuss wider issues.
Tilemaker explicitly states:
But don't use tilemaker if:
- You want the entire planet
Has this concern been addressed, or is there a workaround of sorts that might not be good for others? While we could say that the stack discussion should be postponed, the specific settings to convert download into tiles is stack-specific, so deploying a solution that is not optimal for others could in theory lead to duplicated efforts.
P.S. This is a great news regardless, thank you for working on getting MVT to OSM!
We literally had the author of tilemaker on the operations call on Wednesday to discuss this so yes, we have discussed it.
RAM estimate requirement is between 100GB and 200GB. Estimate time to generate for the full planet is around 4 days runtime on a 2x Intel Xeon X5650 using 144GB RAM system.
@systemed would be able to provide more clarity.
Yes, that's about right; hopefully a bit less time than that. Currently finishing off a set of performance optimisations which will give some harder numbers.
If I understand it well, the goal of the experimental vector tile service is to boost the OSM vector tile style development and to improve the schema for OSM vector tiles - and allow regular iteration on development by the community.
The community around @openmaptiles would be very happy to help with these efforts.
For the upcoming OMT v4 these things may be of interest to OSMF:
- Near real-time updates and serving are technically possible
- An “OSM Carto” type of style - with cartographical goal to demonstrate all included data - developed together with the schema improvements (ideally community PRs to contain both modification of the style snippet and OSM tags / zoom / generalization selection)
- CI tests on every contributed PR - to validate size&speed increase
- Maxzoom 15 in VT schema considered, probably required for a style like OSM Carto, to avoid huge z14 tiles in some places
- Ability to enrich/redefine layers - with different maintainers for separate layers
- All features Include OSM IDs to interact with OSM endpoints - a la https://osmapp.org/
If OSMF is interested - we could provide a weekly updated downloadable tile package for self-hosting and/or live tile endpoint (for proxy via your CDN or direct use on your DNS), based on the latest OSM data and powered by the latest modifications of the schema in GitHub. You could also run the open-source components on-prem with Kubernetes. Just let us know.
Delay on pre-generating an entire planet is about a day - less if materializing only upper zoom levels, with real time on lower zoom levels served from an endpoint, or if only diffs are generated.
Hosting tilemaker as an experiment does not preclude us from hosting other vector tile solutions. This is something we're planning on hosting using resources that will become available later this year. It will not be a featured tile layer on the website to start, and if it should be is a question the OWG will consider at the time. We have in the past hosted experiments that did not end up on osm.org, and looked at multiple approaches for a goal.
For using a live tile endpoint, are you thinking of that on osm.org? We had spoken to someone back in March with the call for featured layers, but communication died out. If you're interested in that, please email information about it and meeting the criteria of the featured layer policy to the OWG.
If you're thinking of instead asking the OWG to host an instance of the OMT stack, please open a new issue so this one remains about maptiler.
Maybe tiles can/should be hosted on vector_tiles.experimental.openstreetmap.org or completely different domain to reduce
keep in mind that when the OSMF rolls out something like that there will be a message of endorsement
effect?
@matkoniecz it is rather clear that what the OSMF runs as its "standard stack" has in the past and likely will in the future have a big effect on the market for such solutions. Just consider, for example, if something else than the openmaptiles schema is chosen (a distinct possibility) for the tiles.
My hope would be that people will start developing a vector style we can use and that there can then be a feedback cycle between that and any necessary changes to the tile schema until we have something we feel is deployable.
The openstreetmap-americana project is presently in the early stages of developing a vector style for the US mapping community, with a small team of active contributors and nearly 100 folks dialed into our project's Slack channel. The focus on our project is to develop a regional style with features of interest to the US community, most notably comprehensive highway shield rendering with concurrencies. This project is currently based on the OpenMapTiles schema. If the OWG-hosted vector server is using an OpenMapTiles configuration, we would be more than happy to collaborate for this purpose.
I'm pleased to report we've managed to perform a one-time planet build of the OpenMapTiles schema at full zoom (z14) using planetiler, and are hosting it on AWS on a temporary basis. I'm providing a description of this test planet build, costs, and technical details in case it is useful for anyone working on cloud vector tile hosting with OpenMapTiles. I also have notes that I took during this process that I'm happy to share with anyone looking to create a similar setup.
The tile server is running here
The key advance of planetiler is the use of in-memory transformations for mbtiles generation. As a result, planetiler is able to execute OpenMapTiles builds orders of magnitude faster than the out-of-the box scripts in the OpenMapTiles distribution. The planet render was done on an AWS m5dn.8xlarge instance, which has 128GB RAM, 32 CPUs, and 2x600GB SSDs (of which we only used 1x600GB). We executed the render on a variably-priced "spot" instance which was running approximately 40 cents per hour at the time of the render. The render took 2h 0m 6s to run on this hardware, plus an additional 6 minutes to transfer the resulting 100GB planet mbtiles file to an AWS Elastic File System (EFS) volume ($10/month). Thus we were able to generate a planet mbtiles file for a bit less than $1. However, this may cost as much as $5 for the same render at peak data center times.
Because there is an OSM planet file hosted on AWS, the download of the planet pbf file takes less than three minutes, so this may result in additional download time on a different hosting provider.
Next, for the tile server we spun up a separate AWS t2.nano instance (0.5GB RAM, 1 CPU) and connected it to the EFS volume, installed docker, and ran tileserver-gl with docker's restart-always setting. The t2.nano instance runs at a cost of $0.0065 per hour (a bit less than $5/month), and costs $0.09 per GB of outbound file transfer.
While this is a low-powered setup, it demonstrates the most basic possible infrastructure that can run a planet vector tile build. An enterprise-level vector tile server would ideally also deploy a CDN and load balancer with higher powered servers in multiple availability zones.
Planetiler is nearly complete, however, it is missing an implementation of OpenMapTiles' street abbreviation functionality. In addition, for this render I turned off wikidata integration, which would have added an additional 15 minutes to the render time.
This setup could be conceptually extended to a cyclic build as follows:
- Download planet
- Update the planet file to latest, for example by running pyosmium-up-to-date
- Render planet, dump to file storage location, restart tile server
- GOTO 2
Just a brief question about zoom levels. Currently the max zoom level supported by each of the osm.org maps are respectively 19, 20, 21, 21, 18 and 20. What would the max zoom level (either technical or practical) of a layer created using either tilemaker , openmaptiles, planetiler or anything else?
I appreciate that with vector tiles you've effectively got "almost infinite overzoom" - you can zoom in as much as you like; but what I'm asking here is "at what level can new features appear" - can they start to be shown at the equivalent of raster zoom 18, 19 or 20?
That's entirely up to the author of the style.
The actual vector tiles normally stop at z14 or so and obviously you're limited by what data your schema includes in the high zoom tiles but beyond that it's up to the style which features in the tile it renders at each zoom level.
Planetiler currently limits to z14 but it should be pretty straightforward to increase if people want to go higher. The z14 limit was to support openmaptiles schema which only needs z14. The main reason to go further is if your style starts including more detail and the z14 tiles get too big. The biggest z14 tiles with the openmaptiles schema are about 1.7mb (before gzip compression) and there are only a handful over 1mb so there wouldn't be much benefit for that schema.
Generating more zoom levels will also make the tile render time longer, and mbtiles file bigger. The most recent run I did on planetiler on a beefy ec2 instance took 47 minutes and rendering z14 took 10 of those minutes, so z15 would probably add at least ~40 minutes and z16 would add at least another ~160 minutes.
It's also worth pointing out that OpenMapTiles omits quite a few features of the "micro mapping" variety that are supported by osm-carto, so if the goal is a vector schema that supports everything that osm-carto does, we could be looking at very substantial mbtiles size inflation over the current 100GB full-zoom mbtiles produced from OpenMaptiles.
I'd suggest to look also at Shortbread https://shortbread.geofabrik.de/ , a lean Vector Tile Schema, which is CC0/FTWPL licensed.
I'd suggest to look also at Shortbread https://shortbread.geofabrik.de/ , a lean Vector Tile Schema, which is CC0/FTWPL licensed.
The challenge at the moment is that somebody needs to develop the tilemaker .lua stylesheet or planetiler extension to support generating this or any other schema in vector tiles. Both tools are able to support OpenMapTiles out of the box, other schemas need developer time.
It would be advisable to keep in mind that most non-trivial cartography, especially at higher zoom levels/larger map scales, requires scale/zoom level dependent geometry processing. And the practical feasibility and efficiency of client side map rendering as it is common today is strongly tied to decidedly not transferring geometry data separately for the higher zoom levels.
I know this issue is not meant to be concerned with actual map design. But the question from @SomeoneElseOSM is pointing in the right direction. The bottom line is if you contemplate the question of choosing a server side data preprocessing framework for client side rendering without regards to the needs of community map design you will likely end up providing a basis exclusively for the the dead end of cartographic primitivism a la mapbox/openmaptiles.
It is no coincidence that many of the most innovative studies in high zoom level OSM based map design done more recently (like https://github.com/SupaplexOSM/strassenraumkarte-neukoelln, https://github.com/enzet/map-machine) are not based on any of the more established tools you are contemplating here.
requires scale/zoom level dependent geometry processing
This is basic functionality in all of the tools we're discussing here, including out-of-the-box openmaptiles.
Link to zoom level specific geometry processing in openmaptiles at z>14 (or any other framework fwiw)?
Geometry generalization is basic functionality in ALL vector tile renderers regardless of which zoom levels you choose to apply it to, whether it's the PostGIS functions used by openmaptiles or the Java Topology Suite used by planetiler, or whatever tilemaker does. It would be helpful if you could provide an example of what you mean by "scale/zoom level dependent geometry processing" because clearly that is done today.
Well - i don't want to get into what would truly be an off-topic discussion here - I was pointing to the need for zoom level specific geometry processing at the high zoom levels, you claimed this to be available routinely in openmaptiles, i was asking for links.
If you wonder about practical use cases - links i gave above show some - as does various stuff on https://github.com/imagico/osm-carto-alternative-colors. But again: This is off-topic here. My point was not discussing specific map design questions, my point was pointing out the inherent limitations of the setups discussed here so far. No one doubts that you can in principle produce a vector tile set with separate tiles for each zoom level up to 24 or higher. But the practical feasibility of all of the client side rendered vector tile systems is based on the fact that this is not done, because - as you said yourself - this would explode the volume of data to be stored and transferred quite massively.
Can we please keep this on topic people.
The goal here is to get a basic set of vector tiles up to encourage the creation of a project to develop a full schema and style that we can use to run a production service.
We don't need to discuss the technical minutiae of what a production quality style might look like at this point!
My aim was specifically not to discuss technical minutiae but to point out strategic issues that could well decide if what is encouraged in terms of style is going to be map design primitivism or avant-garde.