bluet
Freeze after a while when limit not set
After waiting a while when a limit is not set. Proxybroker will stop finding proxies. Is this normal? Shouldn't it find and update proxy lists from providers every time?
@soermejo Thanks for reporting this. Could you provide more detailed info (including the steps we can reproduce)?
I almost always have a stop while processing "proxy-list.org" sometimes on "proxybroker - DEBUG - 98 proxies received from proxy-list.org" or sometimes a little further as in the debug below
` LIMIT_PROXY=8000 judges = ['http://httpbin.org/get?show_env', 'https://httpbin.org/get?show_env' ]
broker = Broker(
proxies, timeout=15, max_conn=300, max_tries=3, verify_ssl=False, judges=judges,
)
tasks = asyncio.gather(
#types=[('HTTP', ('Anonymous', 'High'),'HTTPS')]
broker.find(
types=['HTTPS', 'SOCKS4', 'SOCKS5'],
limit=LIMIT_PROXY
),
save(proxies, collection)
)
`
`
2021-06-24 14:02:19,022 - proxybroker - DEBUG - http://proxy-list.org/english/index.php?p=10 is failed. Error: BadStatusError('Status: 503');
2021-06-24 14:02:19,022 - proxybroker - DEBUG - 0(0) proxies added(received) from http://proxy-list.org/english/index.php?p=10
2021-06-24 14:02:19,023 - proxybroker - DEBUG - 98 proxies received from proxy-list.org: {('180.183.98.144', '8080',
2021-06-24 14:02:23,822 - proxybroker - DEBUG - 139.196.153.156:8080 [HTTPS]: Received: 832 bytes: b'HTTP/1.1 200'; Runtime: 13.42 2021-06-24 14:02:23,822 - proxybroker - DEBUG - 139.196.153.156:8080 [HTTPS]: Get: success; Runtime: 0.00 2021-06-24 14:02:23,822 - proxybroker - DEBUG - 139.196.153.156:8080 [HTTPS]: (https://httpbin.org/get?show_env) rv: 4693, response: b'HTTP/1.1 200 OK\r\nDate: Thu, 24 Jun 2021 11:02:10 GMT\r\nContent-Type: application/json\r\nContent-Length: 607\r\nConnection: close\r\nServer: gunicorn/19.9.0\r\nAccess-Control-Allow-Origin: \r\nAccess-Control-Allow-Credentials: true\r\n\r\n{\n "args": {\n "show_env": ""\n }, \n "headers": {\n "Accept": "/*", \n "Accept-Encoding": "gzip, deflate", \n "Cache-Control": "no-cache", \n "Content-Length": "0", \n "Cookie": "cookie=ok", \n "Host": "httpbin.org", \n "Pragma": "no-cache", \n "Referer": "https://www.google.com/", \n "User-Agent": "PxBroker/0.3.2/4693", \n "X-Amzn-Trace-Id": "Root=1-60d46632-1792b795281215077c0fc0d4", \n "X-Forwarded-For": "139.196.153.156", \n "X-Forwarded-Port": "443", \n "X-Forwarded-Proto": "https"\n }, \n "origin": "139.196.153.156", \n "url": "https://httpbin.org/get?show_env"\n}\n' 2021-06-24 14:02:23,823 - proxybroker - DEBUG - 139.196.153.156:8080 [HTTPS]: Response: correct; Runtime: 0.00 2021-06-24 14:02:23,823 - proxybroker - DEBUG - 139.196.153.156:8080 [HTTPS]: Connection: closed; Runtime: 0.00 2021-06-24 14:02:23,823 - proxybroker - DEBUG - push to result: <Proxy CN 4.80s [HTTPS] 139.196.153.156:8080> Found proxy: <Proxy CN 4.80s [HTTPS] 139.196.153.156:8080> `
