scrapy-proxy-pool icon indicating copy to clipboard operation
scrapy-proxy-pool copied to clipboard

Running spiders sequentially - Proxy Pool is already defined as a collector

Open samLozier opened this issue 4 years ago • 2 comments

I'm trying to run two spiders in sequence in a single file and am getting this error: proxyscrape.errors.CollectorAlreadyDefinedError: proxy-pool is already defined as a collector

I'm attempting to run my spiders like: spider1() spider2()

Either spider will run as expected if I comment out the other line, but they will not run in sequence. Is there an easy work around, or should I separate this behavior into two separate .py files? Editing to add: This is clearly related to the proxyscrape package that proxy-pool is using, but I'm wondering if the behavior triggering the error is coming from my own code, proxy pool or proxyscrape.

Problem: Can't run two spiders in sequence using proxy pool from a .py file Anticipated Results: The second spider would run after the first one concludes.

samLozier avatar Mar 11 '20 18:03 samLozier

Unfortunately I don't have the time to keep this extension up to date. Sorry!

citrus avatar Oct 17 '12 18:10 citrus

Sorry to hear that. Are you aware of an alternative that you think is better?

If I write the updates will you be willing to review/approve pull requests?

Thanks for getting back to me.

Ben-M avatar Oct 17 '12 19:10 Ben-M