django-q2 icon indicating copy to clipboard operation
django-q2 copied to clipboard

Refactor: using task object instead of dict, removing queues to avoid blocking and reorganization of files

Open GDay opened this issue 2 years ago • 1 comments

This is still very much a work in progress! Just saving this here as a draft to track progress. It's pretty much a full rewrite of the library. There is still a lot to do. What has been done so far:

  • Restructured the app into multiple files, to clean up the cluster.py file
  • Replaced shared queues with pipes, which will resolve the blocking issue when a process crashes (fixes #23)
  • Changed the task dict that gets passed through the cluster to a dedicated dataclass, so every task is now an object instead of a dictionary.
  • Task timeouts are now properly caught

It's currently functional for normal async tasks, the rest is still buggy.

What still needs to be done:

  • [x] Fix the scheduling
  • [ ] Redo all tests (basically full rewrite of the test suite)
  • [x] Clean up and change wording of some functions/variables
  • [x] Re-implement monitoring management commands
  • [ ] Probably fixing more things that I will find along the way
  • [ ] Check for breaking changes moving to this version (no breaking changes so far)

Since this is a very heavy rewrite, this will need a lot of eyes before this can be merged (we are not there yet anyway).

GDay avatar Feb 23 '23 01:02 GDay

Pull Request Test Coverage Report for Build 4673412590

  • 588 of 1042 (56.43%) changed or added relevant lines in 21 files are covered.
  • 230 unchanged lines in 14 files lost coverage.
  • Overall coverage decreased (-16.9%) to 73.182%

Changes Missing Coverage Covered Lines Changed/Added Lines %
django_q/exceptions.py 16 17 94.12%
django_q/status.py 0 1 0.0%
django_q/utils.py 5 6 83.33%
django_q/helpers.py 27 34 79.41%
django_q/models.py 23 31 74.19%
django_q/tests/test_cluster.py 66 74 89.19%
django_q/queue_task.py 119 129 92.25%
django_q/management/commands/qinfo.py 38 55 69.09%
django_q/scheduler.py 54 75 72.0%
django_q/process_manager.py 34 57 59.65%
<!-- Total: 588 1042
Files with Coverage Reduction New Missed Lines %
django_q/brokers/init.py 1 89.25%
django_q/core_signing.py 1 80.85%
django_q/monitor.py 1 50.72%
django_q/tests/testing_utilities/multiple_database_routers.py 1 64.71%
django_q/conf.py 2 67.91%
django_q/brokers/redis_broker.py 3 90.2%
django_q/tests/test_commands.py 3 84.21%
django_q/management/commands/qmonitor.py 7 0%
django_q/management/commands/qmemory.py 8 0%
django_q/tests/test_cluster.py 25 90.88%
<!-- Total: 230
Totals Coverage Status
Change from base Build 4589544429: -16.9%
Covered Lines: 2636
Relevant Lines: 3602

💛 - Coveralls

coveralls avatar Apr 11 '23 15:04 coveralls