my8100
refactor(tasks): improve the performance of tasks page
This commit improve the performance of the task page.
Background
When we have millions of tasks log, it will take a significant duration for loading the task page. When loading the task list, process_tasks will load task result for each task items while only the fail_count, pass_count, total, task_id, result_id of the task result are be used. And each of these fields can be computed by database.
Change Code
This commit refactor the process_tasks method in TasksView class, using group by statement and sum, count functions to compute these statistic fields at database.
Performance
This change has been applied in our intranet, and it improve the loading time of task page from 15s to 1s.
Hello @kyy1996! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
- In the file
scrapyd-deploy.py:
Line 15:1: F401 'setuptools' imported but unused Line 28:1: E122 continuation line missing indentation or outdented Line 40:1: E302 expected 2 blank lines, found 1 Line 47:68: E502 the backslash is redundant between brackets Line 49:51: E231 missing whitespace after ',' Line 52:66: E502 the backslash is redundant between brackets Line 60:1: E302 expected 2 blank lines, found 1 Line 87:23: E261 at least two spaces before inline comment Line 97:10: E261 at least two spaces before inline comment Line 111:1: E302 expected 2 blank lines, found 1 Line 126:1: E302 expected 2 blank lines, found 1 Line 129:1: E302 expected 2 blank lines, found 1 Line 133:1: E302 expected 2 blank lines, found 1 Line 141:1: E302 expected 2 blank lines, found 1 Line 147:1: E302 expected 2 blank lines, found 1 Line 152:1: E302 expected 2 blank lines, found 1 Line 165:1: E302 expected 2 blank lines, found 1 Line 171:1: E302 expected 2 blank lines, found 1 Line 174:1: E302 expected 2 blank lines, found 1 Line 199:1: E302 expected 2 blank lines, found 1 Line 218:1: E302 expected 2 blank lines, found 1 Line 222:10: E261 at least two spaces before inline comment Line 230:1: E302 expected 2 blank lines, found 1 Line 252:1: E302 expected 2 blank lines, found 1 Line 267:1: E302 expected 2 blank lines, found 1 Line 278:36: E127 continuation line over-indented for visual indent Line 279:36: E127 continuation line over-indented for visual indent Line 280:36: E127 continuation line over-indented for visual indent Line 281:36: E127 continuation line over-indented for visual indent Line 286:36: E127 continuation line over-indented for visual indent Line 287:36: E127 continuation line over-indented for visual indent Line 288:36: E127 continuation line over-indented for visual indent
- In the file
scrapydweb/run.py:
