ProxyPool icon indicating copy to clipboard operation
ProxyPool copied to clipboard
verified publisher icon Python3WebSpider

run.py 起來後拿不到ＩＰ

Open CubatLin opened this issue 3 years ago • 7 comments

HI想問一下, 透過run.py 啟動後, 用以下python code好像沒辦法成功拿到ＩＰ, 請問要怎麼透過python函式拿代理ＩＰ呢？謝謝

  • python code
>>> import requests
>>> proxypool_url = 'http://127.0.0.1:5555/random'
>>> def get_random_proxy():
...     """
...     get random proxy from proxypool
...     :return: proxy
...     """
...     return requests.get(proxypool_url).text.strip()
... 
>>> get_random_proxy()
  • Terminal logs
(crawler_env) MacBook-Pro ProxyPool-master % python3 run.py                   
2022-05-27 19:30:53.527 | INFO     | proxypool.scheduler:run:100 - starting proxypool...
2022-05-27 19:30:53.527 | INFO     | proxypool.scheduler:run:104 - starting tester, pid None...
2022-05-27 19:30:53.532 | INFO     | proxypool.scheduler:run:110 - starting getter, pid None...
2022-05-27 19:30:53.533 | INFO     | proxypool.scheduler:run:116 - starting server, pid None...
2022-05-27 19:30:53.804 | DEBUG    | proxypool.scheduler:run_tester:33 - tester loop 0 start...
 * Serving Flask app "proxypool.processors.server" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
2022-05-27 19:30:53.804 | INFO     | proxypool.processors.tester:run:74 - stating tester...
2022-05-27 19:30:53.805 | DEBUG    | proxypool.processors.tester:run:76 - 8081 proxies to test
2022-05-27 19:30:53.805 | DEBUG    | proxypool.processors.tester:run:79 - testing proxies use cursor 0, count 20
2022-05-27 19:30:53.806 | DEBUG    | proxypool.processors.tester:test:42 - testing 80.249.135.41:8080
Process Process-3:
Traceback (most recent call last):
  File "/Users/ww/miniforge3/envs/crawler_env/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/ww/miniforge3/envs/crawler_env/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ww/Desktop/Python/Fubon/10. crawler/ProxyPool-master/proxypool/scheduler.py", line 95, in run_server
    app.run(host=API_HOST, port=API_PORT, threaded=API_THREADED)
  File "/Users/ww/miniforge3/envs/crawler_env/lib/python3.8/site-packages/flask/app.py", line 990, in run
    run_simple(host, port, self, **options)
  File "/Users/ww/miniforge3/envs/crawler_env/lib/python3.8/site-packages/werkzeug/serving.py", line 1030, in run_simple
    s.bind(server_address)
OSError: [Errno 48] Address already in use
2022-05-27 19:30:53.831 | DEBUG    | proxypool.processors.tester:test:42 - testing 165.225.196.66:10605
2022-05-27 19:30:53.831 | DEBUG    | proxypool.processors.tester:test:42 - testing 176.235.131.231:9090
2022-05-27 19:30:53.831 | DEBUG    | proxypool.processors.tester:test:42 - testing 199.60.103.241:80
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 187.115.10.50:20183
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 139.5.16.97:8080
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 31.170.174.111:53281
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 8.214.11.10:8118
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 178.22.68.113:8111
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 165.225.194.102:10605
2022-05-27 19:30:53.833 | DEBUG    | proxypool.processors.tester:test:42 - testing 58.240.110.171:8888
2022-05-27 19:30:53.833 | DEBUG    | proxypool.processors.tester:test:42 - testing 110.80.172.180:16790
2022-05-27 19:30:53.833 | DEBUG    | proxypool.processors.tester:test:42 - testing 121.139.171.22:80
2022-05-27 19:30:53.833 | DEBUG    | proxypool.processors.tester:test:42 - testing 36.91.68.149:8080
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 103.139.242.1:83
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 43.241.29.201:8080
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 119.2.41.81:8080
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 78.189.32.215:8080
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 37.110.6.178:8080
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 217.60.194.53:8080
2022-05-27 19:30:54.269 | DEBUG    | proxypool.scheduler:run_getter:48 - getter loop 0 start...
2022-05-27 19:30:54.270 | INFO     | proxypool.processors.getter:run:36 - crawler <public.daili66.Daili66Crawler object at 0x1075b38e0> to get proxy
2022-05-27 19:30:54.271 | INFO     | proxypool.crawlers.base:crawl:40 - fetching http://www.66ip.cn/1.html
2022-05-27 19:30:54.767 | INFO     | proxypool.storages.redis:decrease:84 - 199.60.103.241:80 score decrease 1, current 8.0
2022-05-27 19:30:54.767 | DEBUG    | proxypool.processors.tester:test:65 - proxy 199.60.103.241:80 is invalid, decrease score
2022-05-27 19:30:54.803 | INFO     | proxypool.storages.redis:decrease:84 - 8.214.11.10:8118 score decrease 1, current 9.0
2022-05-27 19:30:54.803 | DEBUG    | proxypool.processors.tester:test:65 - proxy 8.214.11.10:8118 is invalid, decrease score
2022-05-27 19:30:54.816 | INFO     | proxypool.storages.redis:decrease:84 - 110.80.172.180:16790 score decrease 1, current 8.0
2022-05-27 19:30:54.817 | DEBUG    | proxypool.processors.tester:test:65 - proxy 110.80.172.180:16790 is invalid, decrease score
2022-05-27 19:30:55.242 | INFO     | proxypool.storages.redis:decrease:84 - 165.225.194.102:10605 score decrease 1, current 8.0
2022-05-27 19:30:55.243 | DEBUG    | proxypool.processors.tester:test:65 - proxy 165.225.194.102:10605 is invalid, decrease score
2022-05-27 19:30:55.315 | INFO     | proxypool.storages.redis:decrease:84 - 165.225.196.66:10605 score decrease 1, current 5.0
2022-05-27 19:30:55.316 | DEBUG    | proxypool.processors.tester:test:65 - proxy 165.225.196.66:10605 is invalid, decrease score
2022-05-27 19:30:56.255 | INFO     | proxypool.storages.redis:decrease:84 - 121.139.171.22:80 score decrease 1, current 5.0
2022-05-27 19:30:56.256 | DEBUG    | proxypool.processors.tester:test:65 - proxy 121.139.171.22:80 is invalid, decrease score
2022-05-27 19:30:59.110 | ERROR    | proxypool.crawlers.base:crawl:47 - crawler <public.daili66.Daili66Crawler object at 0x1075b38e0> crawled proxy unsuccessfully, please check if target url is valid or network issue
2022-05-27 19:30:59.113 | INFO     | proxypool.processors.getter:run:36 - crawler <public.data5u.Data5UCrawler object at 0x1075b3850> to get proxy
2022-05-27 19:30:59.114 | INFO     | proxypool.crawlers.base:crawl:40 - fetching http://www.data5u.com
2022-05-27 19:31:03.269 | INFO     | proxypool.processors.getter:run:36 - crawler <public.fatezero.FatezeroCrawler object at 0x1075b3a00> to get proxy
2022-05-27 19:31:03.269 | INFO     | proxypool.crawlers.base:crawl:40 - fetching http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.528 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 3.211.17.212:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.529 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 50.16.33.219:8080 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.530 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 54.156.145.160:8080 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.531 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 159.203.127.55:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.531 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 169.57.1.85:8123 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.532 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 132.145.57.226:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.532 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 212.129.29.139:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.533 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 116.203.22.243:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.533 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 46.101.208.13:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.534 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 188.166.166.109:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.535 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 167.99.181.81:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.535 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 177.190.160.55:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.536 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 190.97.226.236:999 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.536 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 190.26.201.194:8080 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.537 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 207.148.97.123:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.537 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 51.91.157.66:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.538 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 13.125.89.72:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.538 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 178.209.51.218:7829 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.539 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 195.135.242.141:8081 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.539 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 167.71.208.92:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.539 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 104.160.189.3:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.540 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 169.57.1.85:8123 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.540 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 190.26.201.194:8080 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.540 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 188.254.0.59:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.541 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 15.165.18.201:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.541 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 122.155.165.191:3128 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.542 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 190.97.226.236:999 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.542 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 80.48.119.28:8080 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.542 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 103.149.162.195:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.543 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 113.252.54.73:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.543 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 49.85.96.80:10081 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.544 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 113.195.127.130:8085 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.544 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 103.232.215.194:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.544 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 223.96.90.216:8085 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.545 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 112.6.117.135:8085 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.545 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 139.255.116.52:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.545 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 112.6.117.178:8085 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.546 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 120.220.220.95:8085 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.546 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 39.175.75.8:30001 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.547 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 178.209.51.218:7829 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.547 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 223.68.190.136:9091 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.547 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 111.3.118.247:30001 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.548 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 95.216.249.203:83 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.548 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 36.7.108.56:9091 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.548 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 181.129.14.165:999 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.549 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 182.92.179.221:443 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.549 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 200.39.153.1:999 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.549 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 164.155.146.31:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.549 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 89.250.152.76:8080 from http://proxylist.fatezero.org/proxy.list
...
...

CubatLin avatar May 27 '22 11:05 CubatLin

想問一下用run.py 啟動後 tester測完能用的ip 要從哪邊透過python code能獲取 謝謝

CubatLin avatar May 27 '22 12:05 CubatLin

你先通过浏览器http://127.0.0.1:5555/random或者http://localhost:5555/random查看能否出现代理ip。看日志，proxypool有在正常运行。

cyicz123 avatar May 28 '22 23:05 cyicz123

運行後，localhost request後沒有收到respond, 沒有ip回傳，想請問有除了random之外能拿到ip的路由嗎

CubatLin avatar May 29 '22 02:05 CubatLin

我也是同样的问题，run之后，http://127.0.0.1:5555/ 这个页面都打不开，一直转圈圈

RuoCJ avatar Feb 15 '23 02:02 RuoCJ

@RuoCJ 用什么方式启动的？Docker 应该没问题

Germey avatar Feb 15 '23 17:02 Germey

@RuoCJ 用什么方式启动的？Docker 应该没问题

没有使用docker，直接用 python run.py

RuoCJ avatar Feb 16 '23 00:02 RuoCJ

@RuoCJ 那估计和环境配置有关系，可以尝试下 Docker 启动。

Germey avatar Feb 19 '23 16:02 Germey