scrapydweb
scrapydweb copied to clipboard
my8100
Feature - Stop long running scrapers.
Introduced a button on the jobs page of scrapydweb, which sends a stop signal to all the currently running scrapers that have been running longer than a certain amount of time, which can be specified in the settings.
This will be especially useful in cases where the project contains hundreds of running spiders.
Hello @knmn2000! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:
- In the file
__init__.py:
- In the file
common.py:
Line 93:13: E722 do not use bare 'except'
- In the file
default_settings.py:
Line 68:121: E501 line too long (158 > 120 characters)
- In the file
models.py:
Line 127:17: E126 continuation line over-indented for hanging indent Line 144:17: E126 continuation line over-indented for hanging indent
- In the file
scrapydweb_settings_v10.py:
Line 68:121: E501 line too long (158 > 120 characters)
- In the file
utils/check_app_config.py:
Line 83:13: W503 line break before binary operator Line 86:13: W503 line break before binary operator Line 87:13: W503 line break before binary operator Line 88:13: W503 line break before binary operator Line 401:13: W503 line break before binary operator Line 617:1: W391 blank line at end of file
- In the file
utils/poll.py:
Line 61:13: E116 unexpected indentation (comment) Line 139:101: E226 missing whitespace around arithmetic operator Line 217:53: E226 missing whitespace around arithmetic operator Line 224:48: E226 missing whitespace around arithmetic operator
- In the file
utils/scheduler.py:
Line 52:5: E116 unexpected indentation (comment) Line 53:5: E116 unexpected indentation (comment) Line 54:5: E116 unexpected indentation (comment)
- In the file
utils/send_email.py:
Line 63:9: E722 do not use bare 'except' Line 66:13: E722 do not use bare 'except' Line 82:13: E722 do not use bare 'except'
- In the file
utils/setup_database.py:
Line 158:9: E722 do not use bare 'except'
- In the file
views/baseview.py:
Line 137:13: W503 line break before binary operator Line 144:13: W503 line break before binary operator Line 512:17: W503 line break before binary operator Line 516:17: W503 line break before binary operator
- In the file
views/dashboard/jobs.py:
Line 150:13: E722 do not use bare 'except' Line 320:25: W503 line break before binary operator
- In the file
views/files/log.py:
Line 68:110: E226 missing whitespace around arithmetic operator Line 146:13: W503 line break before binary operator Line 147:13: W503 line break before binary operator Line 148:13: W503 line break before binary operator Line 148:13: E129 visually indented line with same indent as next logical line Line 269:13: W503 line break before binary operator Line 269:13: E129 visually indented line with same indent as next logical line Line 312:13: E722 do not use bare 'except' Line 371:16: W503 line break before binary operator Line 372:16: W503 line break before binary operator Line 466:20: W503 line break before binary operator Line 494:13: W503 line break before binary operator Line 495:13: W503 line break before binary operator Line 496:9: E124 closing bracket does not match visual indentation
- In the file
views/files/projects.py:
Line 103:13: E722 do not use bare 'except'
- In the file
views/operations/schedule.py:
Line 179:121: E501 line too long (126 > 120 characters) Line 180:121: E501 line too long (129 > 120 characters) Line 181:121: E501 line too long (141 > 120 characters) Line 182:121: E501 line too long (126 > 120 characters)
- In the file
views/operations/scrapyd_deploy.py:
Line 51:4: E114 indentation is not a multiple of four (comment) Line 52:4: E114 indentation is not a multiple of four (comment) Line 53:4: E114 indentation is not a multiple of four (comment) Line 72:5: E722 do not use bare 'except'
- In the file
views/overview/tasks.py:
Line 157:121: E501 line too long (131 > 120 characters) Line 159:121: E501 line too long (122 > 120 characters) Line 160:121: E501 line too long (149 > 120 characters) Line 273:121: E501 line too long (123 > 120 characters)
- In the file
views/system/settings.py:
Line 39:47: E228 missing whitespace around modulo operator
- In the file
views/utilities/send_text.py:
Line 44:44: W503 line break before binary operator Line 45:44: W503 line break before binary operator Line 52:44: W503 line break before binary operator Line 53:44: W503 line break before binary operator Line 56:44: W503 line break before binary operator Line 57:44: W503 line break before binary operator
I don't think this should be handled by ScrapydWeb. Scrapy has a extension scrapy.extensions.closespider.CloseSpider to close spiders on specific condition. Setting CLOSESPIDER_TIMEOUT is what you want.
