scrapy-proxy-pool

scrapy-proxy-pool copied to clipboard

I am new to scrapy, I have a very large data set wanna scrape so that I need to using multiple thread like spider to do the crawling so that...

seanng123456789

WARNING: No proxies available.

3

Recently I have observed that the below-mentioned message is always showing. That means it is always trying to download with the host IP! Why is this happening? ``` 2022-06-10 18:50:54...

raisulrana

how to use proxies.txt

1

I have list of 250 odd proxies and want to set them like PROXIES_LIST but I guess scrapy-proxy-pool dosent work in that fashion like plain python. How can i make...

vijaykurhade

How do i use my own proxies list?

4

I cannot find anything in the docs to use my own proxy. I have an api that returns me bunch of proxies.

superryeti

Response content isn't text

7

When I add PROXY_POOL_ENABLED = True and DOWNLOADER_MIDDLEWARES = { # ... 'scrapy_proxy_pool.middlewares.ProxyPoolMiddleware': 610, 'scrapy_proxy_pool.middlewares.BanDetectionMiddleware': 620, # ... } to my settings.py file I am encountering the following error: AttributeError:...

m1ngle

Running spiders sequentially - Proxy Pool is already defined as a collector

2

I'm trying to run two spiders in sequence in a single file and am getting this error: `proxyscrape.errors.CollectorAlreadyDefinedError: proxy-pool is already defined as a collector` I'm attempting to run my...

samLozier

A lot of proxies failing

3

2019-10-26 13:42:37 [scrapy_proxy_pool.middlewares] DEBUG: Retrying with another proxy (failed 1 times, max retries: 5) 2019-10-26 13:42:37 [scrapy_proxy_pool.middlewares] DEBUG: [ProxyChoosen] http://45.76.6.67:8888 2019-10-26 13:42:39 [scrapy_proxy_pool.middlewares] DEBUG: Retrying with another proxy (failed 1...

mapb1994

This is a very weird behaviour since either none of the websites have a HTTPS proxy, which is not possible or a list of country codes for PROXY_POOL_FILTER_CODE is not...

usdq777

IndexError: list index out of range

5

Hello, I'm encountering this error lately: File "/Users/mac/PycharmProjects/scrapy/venv/lib/python3.7/site-packages/proxyscrape/scrapers.py", line 164, in get_proxy_daily_http_proxies return _get_proxy_daily_proxies_parse_inner(centers[0], 'http', 'proxy-daily-http') IndexError: list index out of range any help please?

protonhs