infrastructure
infrastructure copied to clipboard
Refactor workers into a queue system
Instead of having all workers poll from typically the same data set, separate them into publishers and consumers via a network queue. Publishers take third-party APIs (DigitalOcean, Github, etc.) and publish their JSON data for consumers to react and change states.
Potentially keep a database cache of all the recent JSON objects (Redis, Mongo, or maybe SQL?)