actor-templates
actor-templates copied to clipboard
Scrapy template failure can pass tests
In situation where scrapy template explodes and produces no results, template test passes. It is because test framework checks only for return code of template script and the scrapy code from template swallows all the exceptions. So even in case of such critical failure the template script returns status code 0
Adapt the test(or the template?) to make it fail is such scenarios.
Error example:
Unhandled error in Deferred:
[twisted] CRITICAL Unhandled error in Deferred:
Traceback (most recent call last):
File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\twisted\internet\defer.py", line 2017, in _inlineCallbacks
result = context.run(gen.send, result)
File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\scrapy\crawler.py", line 154, in crawl
yield self.engine.open_spider(self.spider, start_requests)
File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\twisted\internet\defer.py", line 2017, in _inlineCallbacks
result = context.run(gen.send, result)
File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\scrapy\core\engine.py", line 393, in open_spider
if d := scheduler.open(spider):
File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\apify\scrapy\scheduler.py", line 59, in open
self._rq = nested_event_loop.run_until_complete(open_queue())
File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\base_events.py", line 623, in run_until_complete
self._check_running()
File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\base_events.py", line 585, in _check_running
raise RuntimeError(
builtins.RuntimeError: Cannot run the event loop while another loop is running
No results example:
[scrapy.statscollectors] INFO Dumping Scrapy stats:
{'items_per_minute': None,
'log_count/ERROR': 2,
'log_count/INFO': 8,
'log_count/WARNING': 1,
'responses_per_minute': None}