Exceptions a-plenty, early points get scanned repeatedly, late ones sometimes miss out

[c00ni]

There are several exceptions being thrown, mainly from the pgoapi, some from other sources. Plenty has changed and there are sure to be teething issues. That's fine.

But when a worker restarts following an exception, it starts from the beginning of it's point list and it seems almost up to chance that a point near the end of it's points will ever get scanned in any given 15minute period due to the worker maybe restarting.

This means some points get scanned very many times and some will get scanned much less frequently.

Is it possible to make restarted workers start scanning from the last scanned point (or maybe from the next point to prevent possible loops) to even out the scanning even in the presence of exceptions?

[c00ni]

A stack (almost all) of the errors were database errors. I moved to MySQL, increased the pool size and stopped logging forts and they have all gone away.

I've got one worker stuck, seemingly at one particular cell. It happens every few minutes and the worker is forced to restart. It's the only recurrent error in my log now.

[2016-08-09 19:04:18,250][ worker-32][ ERROR][L 138] A wild exception appeared! Traceback (most recent call last): File "worker.py", line 133, in run self.main() File "worker.py", line 174, in main map_objects = response_dict['responses'].get('GET_MAP_OBJECTS', {}) TypeError: 'NoneType' object has no attribute '__getitem__'

[Aiyubi]

run pip install -r requirements.txt --upgrade

[modrzew]

I implemented restarting worker from its last point in coroutines branch. I have a feeling that it will be merged to master soon.