Add "serveAllData" option

[r0hitsharma]

This is based on the serveAllStyles implementation in https://github.com/maptiler/tileserver-gl/commit/d7a34f3a74bf41de8343367bcefce2f0e0a06539. This does the same for the mbtiles directory.

@petrsloup maybe serveAllTiles is a better name for this option? I can't choose between the two.

[abillits]

We've been testing this and haven't found any issues. Appears to work just fine even on our servers that server over 1k different .mbtiles.

[durkie]

i would love to see this merged into master as well -- would like to serve over 10k mbtiles files.

[candux]

This would be very useful. It works very good to serve pure mbtiles. But I wasn't able to reference a loaded mbtile in a style like usual

    "openmaptiles": {
      "type": "vector",
      "url": "mbtiles://planet.mbtiles"
    },

do I have to use another syntax or is this just not possible?

[xabbu42]

The problem of @candux stems from the fact that the original pull requests does not update the necessary array containing all data sources. I fixed that (and a copy paste error which made that the filesystem watcher never worked) in https://github.com/xabbu42/tileserver-gl/tree/watch-tiles.

I don't think this or my branch is ready to be merged, as the interaction of serveAllStyles and serveAllData is not well defined. Removing a mbtile can invalidate a style that uses it, but the style is kept around with server-side rendering still working but not client-side (due to the fact that the mbtiles is opened separately for server-side rendering). On the other hand there is a race when copying styles and mbtiles together as the style does not get added if the mbtile is still missing.

[pascal-codetaal]

@xabbu42 Can we help with getting this released? We are able to put work hours into this feature.

[xabbu42]

Its not my call to merge this and I can't help much either as we currently do not use tileserver-gl. In my opinion whats missing is updating the list of active styles when new mbtiles are added or removed. That is I would expect a style which was not loaded/activated because of a missing mbtile to load/activate when the mbtile comes available and to unload/deactivate when it gets removed. That way there should be no races and the state of the server is not dependent on the order files get added or removed.

But as I said this is just my 2 cents and I have no say if this is necessary and/or sufficient for merging.

[acalcutt]

I would consider merging something like this, it does sounds like a good feature. but that interaction with styles does sound like it needs some work. It's a bit complicated though if you are adding styles and data on the fly....when styles don't usually show when their data is missing....

For the 'watcher' part of this... In my project I have an several mbtiles files I update that always have the same name. Would this recognize that this is a new file? Is there actually code that removes the file right now or does it just watch for new files....maybe I missed that looking through the code.