Make RedisWQ a separate package and test it thoroughly

Open vringar opened this issue 5 years ago • 2 comments

The work queue we use for the crawler isn't specific to OpenWPM and can be useful to other projects. Also by pulling it out we have an easier time testing it.

Dec 01 '20 10:12 vringar

We know that the Work Queue is currently broken but don't have the time to figure out how and why. This would be one of the first issues in the new repository.

Dec 01 '20 10:12 vringar

Currently some jobs get stuck assigned to an instance that has crashed which means they won't get retried.

Dec 04 '20 16:12 vringar