[New Scheduler] Feature Request for etcd action duration checker

[bdoyle0182]

I think this would be better than the current duration checker method to query elasticsearch over a 24 hour window and removes the requirement for you to be using elasticsearch to fully leverage the benefit of the new scheduler.

My thought is you could easily just store a key value for the average duration in etcd with a lease of 24 hours. And it's updated from the memory queue for the action every time it works through the buffer size. i.e. if the buffer size is 10, on every 10th new activation re-write the average action duration based on the current buffer average. I think you could just make a call from MemoryQueue to DataManagementService though I'm not sure how the leasing would be handled, and then you would have to write a duration checker to lookup the key value from data management service when the memory queue is restarted.

This should be both more optimal resource wise in that you just store a single key / value and don't have to run an expensive query , reusable regardless of whatever activation backend you are using, and should be a better heuristic with the most up to date data when the queue restarts