plugin-sdk
plugin-sdk copied to clipboard
feat: Add queue based scheduler
Summary
Mostly an experiment to deal with https://github.com/cloudquery/cloudquery-issues/issues/2227 as I couldn't think of a nice way to make singleNestedTableMaxConcurrency dynamic without making the code super complex.
This PR adds a scheduler that uses a worker pool pattern on top of a priority queue. This should ensure that as long as there's work to be done, all Go routines will be occupied. Also the concurrency setting is not only for the top level tables, as it's the number of workers so it's a fixed limit and simpler.
The more table client pairs in the queue the less priority they'll have, this should prevent a specific table from occupying all the workers.
Opening as draft since:
- I'm still testing this to see the impact
- There's still a lot of code duplication with current code and refactoring needed to avoid it
Use the following steps to ensure your PR is ready to be reviewed
- [ ] Read the contribution guidelines ๐งโ๐
- [ ] Run
go fmtto format your code ๐ - [ ] Lint your changes via
golangci-lint run๐จ (install golangci-lint here) - [ ] Update or add tests ๐งช
- [ ] Ensure the status checks below are successful โ