actor-templates icon indicating copy to clipboard operation
actor-templates copied to clipboard

Scrapy template failure can pass tests

Open Pijukatel opened this issue 11 months ago • 0 comments

In situation where scrapy template explodes and produces no results, template test passes. It is because test framework checks only for return code of template script and the scrapy code from template swallows all the exceptions. So even in case of such critical failure the template script returns status code 0

Adapt the test(or the template?) to make it fail is such scenarios.

Error example:

Unhandled error in Deferred:
[twisted] CRITICAL Unhandled error in Deferred:

Traceback (most recent call last):
  File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\twisted\internet\defer.py", line 2017, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\scrapy\crawler.py", line 154, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\twisted\internet\defer.py", line 2017, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\scrapy\core\engine.py", line 393, in open_spider
    if d := scheduler.open(spider):
  File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\apify\scrapy\scheduler.py", line 59, in open
    self._rq = nested_event_loop.run_until_complete(open_queue())
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\base_events.py", line 623, in run_until_complete
    self._check_running()
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\base_events.py", line 585, in _check_running
    raise RuntimeError(
builtins.RuntimeError: Cannot run the event loop while another loop is running

No results example:

[scrapy.statscollectors] INFO  Dumping Scrapy stats:
{'items_per_minute': None,
 'log_count/ERROR': 2,
 'log_count/INFO': 8,
 'log_count/WARNING': 1,
 'responses_per_minute': None} 

Pijukatel avatar Jan 14 '25 13:01 Pijukatel