worker icon indicating copy to clipboard operation
worker copied to clipboard

feat(perf): significantly improve "large jobs table" performance characteristics

Open benjie opened this issue 2 years ago • 0 comments

Description

When the jobs table gets large, Graphile Worker gets slower and slower. This is not ideal; it can happen:

  • if you have jobs that keep failing
  • if you add a lot of jobs to a single named queue
  • if you schedule a lot of jobs in the future
  • if you queue jobs into the database that one or more of your instances do not support

This PR addresses this for a few scenarios, significantly improving the "performance floor" - for some real world scenarios by a factor of 20x!

TODO:

  • [ ] Move resetLocked from Worker to Pool (only need one per database, not one per worker)
  • [ ] Add events for resetLocked: start, success, failure

Performance impact

Significant improvement for worst case scenario. Slight improvement for normal usage.

Security impact

None known.

Checklist

  • [x] My code matches the project's code style and yarn lint:fix passes.
  • [ ] ~~I've added tests for the new feature, and yarn test passes.~~
  • [ ] ~~I have detailed the new feature in the relevant documentation.~~
  • [x] I have added this feature to 'Pending' in the RELEASE_NOTES.md file (if one exists).
  • [x] If this is a breaking change I've explained why.

benjie avatar Jun 30 '22 15:06 benjie