scrapy-webdriver
scrapy-webdriver copied to clipboard
added the process_spider_exception method in middleware
To address the issue in Issue #4 of a single exception in a Spider's parse method causing the spider to finish.
Take a look at scrapy_webdriver.tests.test_request_queue. One way to do it would be to base the test on TestRequestQueue.test_priorization, have the mock raise an exception midway, and make sure subsequent requests do get called in addition to somehow test the handling of the exception itself.