Rethink how jobs are managed

[edgurgel]

Right now the WorkersManager does a couple of tasks:

• Ask for jobs to be done through the QueueManager;
• Message the workers to perform a job;
• Receive feedback from workers related to a job (done or failure messages);
• Track jobs that are being done in an ETS table;
• Monitor workers so we know if a job failed because a worker simply died;
• Acknowledge or schedule for retry through the QueueManager;
• Clean up the inprocess list of jobs that were hanging from previous execution

This issue is a starting point to discuss what and if we can split these concerns in different processes but keeping the same feature set we have now.

cc/ @mitchellhenke

[mitchellhenke]

My initial thought is that I could imagine seeing them in a couple groups (maybe even more?):

Worker/Job Management Process:

• Receive feedback from workers related to a job (done or failure messages);
• Track jobs that are being done in an ETS table;
• Monitor workers so we know if a job failed because a worker simply died;
• Acknowledge or schedule for retry through the QueueManager;

Job Queuing:

• Ask for jobs to be done through the QueueManager;
• Message the workers to perform a job;
• Clean up the inprocess list of jobs that were hanging from previous execution;

Appreciate the inclusion, thank you :)