proxy_fetcher icon indicating copy to clipboard operation
proxy_fetcher copied to clipboard

[WIP] Add new proxy providers

Open ydakuka opened this issue 3 years ago • 6 comments

ydakuka avatar May 15 '21 10:05 ydakuka

Love this :heart:

nbulaj avatar Jun 10 '21 15:06 nbulaj

Oh sorry, I still want to finish this. There is not enough time for the last month.

ydakuka avatar Jun 10 '21 15:06 ydakuka

Need this PR to be merged quicker, because now only working provider is force_proxy_list_ssl

my ISP: "МТС",
test from console (-p each_provider >> proxies.txt)
{ expired: %i[free_proxy_list proxyscrape_http proxypedia],
  0: %i[free_proxy_list_socks free_proxy_list_us http_tunnel mtpro proxy_list proxyscrape_socks4 proxyscrape_socks5 xroxy] }

@ydakuka, please megre (ruby3 related) last main changes to your branch: new-proxy-providers.

JuPlutonic avatar Aug 12 '22 10:08 JuPlutonic

FYI, results of checking changed providers (new/xpath/naming/pagination). When proxies are scrapped, they added to proxies.txt with wrong prefix //:

~checker_proxy~
W, [2022-08-12T14:35:48.087251 #14347]  WARN -- : Failed to process request to https://checkerproxy.net/api/archive/2022-08-12 (error reading from socket: Could not parse data)
~f_p_cz~
//208.52.137.X:P
//208.52.137.X:P
//208.52.137.X:P
//208.52.180.X:P
~f_p_l~
0 – nothing – maybe 499 Client Closed Request
~f_p_l_socks~
W, [2022-08-12T14:39:33.114649 #24986]  WARN -- : Failed to process request to https://www.socks-proxy.net/ (error reading from socket: Could not parse data)
~f_p_l_us~
0 – nothing
~proxy_list~
//8.210.155.X:P
//103.252.1.X:P
~proxy_list_download_http~
0 – nothing
~proxy_list_download_https~
0 – nothing
~proxy_list_download_socks4~
0 – nothing
~proxy_list_download_socks5~
0 – nothing
~proxyscrape_http~
//85.14.243.X:P
//89.208.219.X:P
//121.156.109.X:P
//103.252.1.X:P
//81.23.100.X:P
~proxyscrape_socks4~
0 – nothing
~proxyscrape_socks5~
0 – nothing
~scapingant~
0 – nothing
~xroxy~ 
hangs/ infinite processing

In TUI-standalone-mode result = 11. In ProxyFetcher::Client, I think, result would be 10 times greater, and at last better result would be with own client implementation around headless chrome (Ferrum(capybara+cuprite / vessel)).

JuPlutonic avatar Aug 12 '22 14:08 JuPlutonic

@JuPlutonic I don't work with it anymore. Feel free to complete MR.

ydakuka avatar Aug 14 '22 10:08 ydakuka

Ok, the PR/MR will be tested after the 0.16.0/main rebase on it and after recheck. App with the ProxyFetcher::Client will be run and logged with providers mentioned above with ISPs: МТС/another.

JuPlutonic avatar Aug 14 '22 11:08 JuPlutonic