add random variation to cache expiry to distribute cache refreshes

[stephankn]

After expensive mapillary operations got enabled the processing time of an analyzer is about 4-5 times higher than before. This is due to building the cache from external web sources. With the current caching system all of my 120 analyzers will expire on the same day and explode again. This will happen then in multiples of the cache duration.

I think cache expiry is handled here: https://github.com/osm-fr/osmose-backend/blob/c3fa8c3815955b60f49a0b60fbf0b467106a2f81/modules/downloader.py#L93

I propose to add an random element here to expire some content earlier. So cache duration would specify the maximum cache duration.

Code could be somethin like the following which expires a file up to four days (100 hours) earlier, on average 2 days. Typically analyzers are only running once a day, so this would result in an effective stepping of 24 hours. specifying like this would allow analysers to run more frequently and still spread the cache update.

if (cur_time - delay*24*60*60 - random.randrange(0, 100, 1)*60*60) < statbuf.st_mtime:

[frodrigo]

@jocelynj already proposed to do think like this. But the cache already take days (weeks) to fill. So, I think expiration will spread them self in time. cc @cbeddow

[stephankn]

I managed to fill the cache within 3 days having multiple instances running in parallel. As Osmose analysis breaks once a complete cycle takes more than 24 hours, I would like to spread it out a bit beforehand. Otherwise it will always fail in multiples of 30 days. It is not about spreading the load on Facebook, but spreading the load on the analysis machine.

I am doing now manual cache expiry until somethin is available in the code. This removes all cached entries in just less than 30 days. Randomly picks 8 out of the oldest 25 cache entries and removes them.

grep -Zl mapillary *.url | xargs -0 -r stat --format='%Y+%n' | sort -t+ -k 1,1n | head -25 | sort -R | head -8 | cut -d+ -f2- | xargs -I{} basename {} .url | xargs -I{} sh -c "rm {}*"

[stephankn]

looks like it is 60 days. Does not help much. I reduced my expiry workaround to 4 out of oldest 12.

https://github.com/osm-fr/osmose-backend/blob/c3fa8c3815955b60f49a0b60fbf0b467106a2f81/analysers/Analyser_Merge_Mapillary.py#L53

[cbeddow]

Is this not a Mapillary/Facebook issue then, or can you see that the requests are being throttled still? It will throttle based on IP address to prevent DDOS but there might be special permissions available

[stephankn]

Hello @cbeddow , the issue here is that updating the Mapillary data is a quite slow task. My server runs about 120 analyzers. The total executing time is ok, it will finish the analysis within one day. But as soon as caches are updated, the processing duration is about 4-5 times longer on each analyzer.

As they all started at the same day, the caches will also expire all on the same day. So I have a peak where analyzers fail to complete for 4-5 days, then about two months working fine and then they are all taking excessive time again, all on the same timeframe.

Situation gets worse if the cache update would be triggered by a SW change in osmose. Then not only my analyzers start requesting data on that day, but all others as well.

Idea was to spread out the updates. So on each day only a hand-full of of the analyzers is updating the caches. So instead of peaks and waves there is a constant update going on.

[frodrigo]

We will try to setup a proxy to access to Mapillary data. The proxy will have a better access to the Mapillary tiles.

@cbeddow we will recontact you once we setup it.

[stephankn]

@frodrigo having a cache with direct access, would likely also resolve the concerns I raised in #1460 If it is then even faster, having peaks to download are no more an issue, making it obsolete to spread out downloads over time. Would potentially also not hurt, but not as urgent any more.

[frodrigo]

The centralized cache to acces Mapillary tiles is now generalized to all Osmose backend.