scrapy-proxy-pool
scrapy-proxy-pool copied to clipboard
Running spiders sequentially - Proxy Pool is already defined as a collector
I'm trying to run two spiders in sequence in a single file and am getting this error:
proxyscrape.errors.CollectorAlreadyDefinedError: proxy-pool is already defined as a collector
I'm attempting to run my spiders like:
spider1()
spider2()
Either spider will run as expected if I comment out the other line, but they will not run in sequence. Is there an easy work around, or should I separate this behavior into two separate .py files? Editing to add: This is clearly related to the proxyscrape package that proxy-pool is using, but I'm wondering if the behavior triggering the error is coming from my own code, proxy pool or proxyscrape.
Problem: Can't run two spiders in sequence using proxy pool from a .py file Anticipated Results: The second spider would run after the first one concludes.
Unfortunately I don't have the time to keep this extension up to date. Sorry!
Sorry to hear that. Are you aware of an alternative that you think is better?
If I write the updates will you be willing to review/approve pull requests?
Thanks for getting back to me.