worker
worker copied to clipboard
feat(perf): significantly improve "large jobs table" performance characteristics
Description
When the jobs table gets large, Graphile Worker gets slower and slower. This is not ideal; it can happen:
- if you have jobs that keep failing
- if you add a lot of jobs to a single named queue
- if you schedule a lot of jobs in the future
- if you queue jobs into the database that one or more of your instances do not support
This PR addresses this for a few scenarios, significantly improving the "performance floor" - for some real world scenarios by a factor of 20x!
TODO:
- [ ] Move
resetLocked
from Worker to Pool (only need one per database, not one per worker) - [ ] Add events for
resetLocked
:start
,success
,failure
Performance impact
Significant improvement for worst case scenario. Slight improvement for normal usage.
Security impact
None known.
Checklist
- [x] My code matches the project's code style and
yarn lint:fix
passes. - [ ] ~~I've added tests for the new feature, and
yarn test
passes.~~ - [ ] ~~I have detailed the new feature in the relevant documentation.~~
- [x] I have added this feature to 'Pending' in the
RELEASE_NOTES.md
file (if one exists). - [x] If this is a breaking change I've explained why.