Feature Request - S3 cache - MetaTiles

[dBitech]

Can we look to support the idea of Metatiles for the cache. This could minimize what would otherwise be billions of individual files, Amazon S3 GET pricing in us-east is currently about 1/2 cent per 10,000 requests. If the cache could be improved to make use of these MetaTiles, the number of IO requests could be cut down extensively. Meta title support could also not just be limited to S3. File Cache overhead could likely be reduced as well for may operations by not having the very large number of small files in a directory, especially if the underlying FS has an 8 or even 32K block size.

[ARolek]

@dBitech I'm open to ideas. I have not implemented meta titles but I have read up on them a bit. A couple of articles from @mojodna:

• https://medium.com/@mojodna/tapalcatl-cloud-optimized-tile-archives-1db8d4577d92
• https://medium.com/@mojodna/tapalcatl-2-updates-8eb827d969c1

I would need to think through this a bit to see how it could be implemented. One of the benefits of the current S3 approach is that it static hosting can be turned on and a CDN can read directly from it. Some of this architecture is discussed in a post I published recently.

Maybe this could be a config option for the S3 cache? Have you implemented MetaTiles?

[dBitech]

I understand the simpleness and cleanliness of serving direct from S3 with CDN caching but does not necessarily fit well for all use cases. this is not to say that the MetaTile option is a bee all end-all solution. I agree that if MetaTiles are supported, it should be a config option. I've dabbled in using tapalcatl https://github.com/tilezen/tapalcatl as a meta tile server, but it still requires something to produce those MetaTiles.

[ARolek]

@dBitech let's keep the conversation open. I still need to do some research on this as it's not a concept I'm familiar with. Are there other metaTile implementations you're aware of?

[dBitech]

a number of tile backends, (Servers, Rendrers, Caching chains) implement metatiles. TileStache, Mapserver, Mod_tile, Tirex ..... though most appear to target raster tiles. As you can see from the @mojodna articles, that the concept of random access in zip files is straight forward and can work with any content type.

[tierpod]

Hello. I've read this issue and I have to say that metatiles format from tapalcatl articles is different from metatiles format that described in OSM wiki (renderd + mod_tile, tirex, ...). If you are interested in osm metatiles implementation, you can take a look to my repo - feel free to use it if you want.

[ARolek]

@tierpod thanks for chiming in. I will give it a read soon and circle back with any questions. This feature is not currently on our immediate list of priorities but if someone wants to tackle it I'm happy to help with reviewing the PR.

[tierpod]

@ARolek I'm happy to help to the tegola project, especially we are using metatiles cache in our company. And I thought a couple times about how to implement metatiles cache in tegola. But as I understand, cache interface Set method is acting on one single tile (where val []byte is the bytes data of specific tile):

https://github.com/go-spatial/tegola/blob/68db716d8dff79eccc62af131cfd26866ebe1ad7/cache/cache.go#L17-L21

But metatiles format from renderd+mod_tile / tirex / etc projects it is the binary file that contains N*N tiles (8*8 tiles by default) and there is no way to add ("set") one tile to metatile file. We have to have all 8*8 tiles, calculate each tile's size, and write all these data to one metatile file.

In a nutshell, metatile file with size = 8 (64 tiles in metatile file) contains:

metatile file header +
64 tile's binary data offset +
64 tile's binary data

So, perhaps it will require a big changes in cache interface and rendering behaviour to implement this. I'll try to read tegola sources and think about this again.