optimize mapillary object download

[stephankn]

Downloading data from mapillary for analysers to merge objects/street signs takes excessively long. For example china_heilongjiang fetches over 170.000 tiles for each analyser. Tibet 250k. Even with 4-5 requests per second this takes quite long.

I suppose this is also not in the interest of Facebook/Mapillary to have their servers flooded with millions of requests.

How about asking for a special, potentially more private endpoint on the API which returns larger areas on each request? This data might be preprocessed by some batch process on mapillary side, Maybe could aggregate data from z14 levels into something more compact. Data does not need to be "live" date. Some preprocessed data is fine. maybe on z8 level? That would reduce the number of requests by a factor of 4096. When reading z6 aggregation it would be a factor of 65.536. As processing of data happens on server side and not on browser javascript, size is not so much a concern.

I assume that with Osmose covering the full land-mass of the planet and doing refreshes every 30 days, it is by far the highest contributor to their API traffic here.

There is likely already some exception in place for osmose, as typical API rate limit on tiles is 50k. https://www.mapillary.com/developer/api-documentation/#rate-limits

Tiles are accessed on z14. I estimate, that out of the theoretical 270M, Osmose reads 105M tiles. That would be 3.5M tiles per day. Multiplied by two for the two analyzers results in an average request rate of 7,000,000 tiles per day (vs. an API limit of 50k),

[frodrigo]

All limitation are on Facebook side. I already talk about this with @cbeddow. But at this point nothing special has be done.

Note, the cache is configured for 60 days but we may made it logger in regard on how hard it is to fill it.