plugin-sdk icon indicating copy to clipboard operation
plugin-sdk copied to clipboard

feat: Add queue based scheduler

Open erezrokah opened this issue 1 year ago โ€ข 0 comments

Summary

Mostly an experiment to deal with https://github.com/cloudquery/cloudquery-issues/issues/2227 as I couldn't think of a nice way to make singleNestedTableMaxConcurrency dynamic without making the code super complex.

This PR adds a scheduler that uses a worker pool pattern on top of a priority queue. This should ensure that as long as there's work to be done, all Go routines will be occupied. Also the concurrency setting is not only for the top level tables, as it's the number of workers so it's a fixed limit and simpler. The more table client pairs in the queue the less priority they'll have, this should prevent a specific table from occupying all the workers.

Opening as draft since:

  1. I'm still testing this to see the impact
  2. There's still a lot of code duplication with current code and refactoring needed to avoid it

Use the following steps to ensure your PR is ready to be reviewed

  • [ ] Read the contribution guidelines ๐Ÿง‘โ€๐ŸŽ“
  • [ ] Run go fmt to format your code ๐Ÿ–Š
  • [ ] Lint your changes via golangci-lint run ๐Ÿšจ (install golangci-lint here)
  • [ ] Update or add tests ๐Ÿงช
  • [ ] Ensure the status checks below are successful โœ…

erezrokah avatar Aug 14 '24 15:08 erezrokah