django-q2
django-q2 copied to clipboard
Refactor: using task object instead of dict, removing queues to avoid blocking and reorganization of files
This is still very much a work in progress! Just saving this here as a draft to track progress. It's pretty much a full rewrite of the library. There is still a lot to do. What has been done so far:
- Restructured the app into multiple files, to clean up the
cluster.py
file - Replaced shared queues with pipes, which will resolve the blocking issue when a process crashes (fixes #23)
- Changed the task dict that gets passed through the cluster to a dedicated dataclass, so every task is now an object instead of a dictionary.
- Task timeouts are now properly caught
It's currently functional for normal async tasks, the rest is still buggy.
What still needs to be done:
- [x] Fix the scheduling
- [ ] Redo all tests (basically full rewrite of the test suite)
- [x] Clean up and change wording of some functions/variables
- [x] Re-implement monitoring management commands
- [ ] Probably fixing more things that I will find along the way
- [ ] Check for breaking changes moving to this version (no breaking changes so far)
Since this is a very heavy rewrite, this will need a lot of eyes before this can be merged (we are not there yet anyway).
Pull Request Test Coverage Report for Build 4673412590
- 588 of 1042 (56.43%) changed or added relevant lines in 21 files are covered.
- 230 unchanged lines in 14 files lost coverage.
- Overall coverage decreased (-16.9%) to 73.182%
Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
---|---|---|---|
django_q/exceptions.py | 16 | 17 | 94.12% |
django_q/status.py | 0 | 1 | 0.0% |
django_q/utils.py | 5 | 6 | 83.33% |
django_q/helpers.py | 27 | 34 | 79.41% |
django_q/models.py | 23 | 31 | 74.19% |
django_q/tests/test_cluster.py | 66 | 74 | 89.19% |
django_q/queue_task.py | 119 | 129 | 92.25% |
django_q/management/commands/qinfo.py | 38 | 55 | 69.09% |
django_q/scheduler.py | 54 | 75 | 72.0% |
django_q/process_manager.py | 34 | 57 | 59.65% |
<!-- | Total: | 588 | 1042 |
Files with Coverage Reduction | New Missed Lines | % |
---|---|---|
django_q/brokers/init.py | 1 | 89.25% |
django_q/core_signing.py | 1 | 80.85% |
django_q/monitor.py | 1 | 50.72% |
django_q/tests/testing_utilities/multiple_database_routers.py | 1 | 64.71% |
django_q/conf.py | 2 | 67.91% |
django_q/brokers/redis_broker.py | 3 | 90.2% |
django_q/tests/test_commands.py | 3 | 84.21% |
django_q/management/commands/qmonitor.py | 7 | 0% |
django_q/management/commands/qmemory.py | 8 | 0% |
django_q/tests/test_cluster.py | 25 | 90.88% |
<!-- | Total: | 230 |
Totals | |
---|---|
Change from base Build 4589544429: | -16.9% |
Covered Lines: | 2636 |
Relevant Lines: | 3602 |