proxy_fetcher
proxy_fetcher copied to clipboard
[WIP] Add new proxy providers
Love this :heart:
Oh sorry, I still want to finish this. There is not enough time for the last month.
Need this PR to be merged quicker, because now only working provider is force_proxy_list_ssl
my ISP: "МТС",
test from console (-p each_provider >> proxies.txt)
{ expired: %i[free_proxy_list proxyscrape_http proxypedia],
0: %i[free_proxy_list_socks free_proxy_list_us http_tunnel mtpro proxy_list proxyscrape_socks4 proxyscrape_socks5 xroxy] }
@ydakuka, please megre (ruby3 related) last main changes to your branch: new-proxy-providers.
FYI, results of checking changed providers (new/xpath/naming/pagination).
When proxies are scrapped, they added to proxies.txt with wrong prefix //
:
~checker_proxy~
W, [2022-08-12T14:35:48.087251 #14347] WARN -- : Failed to process request to https://checkerproxy.net/api/archive/2022-08-12 (error reading from socket: Could not parse data)
~f_p_cz~
//208.52.137.X:P
//208.52.137.X:P
//208.52.137.X:P
//208.52.180.X:P
~f_p_l~
0 – nothing – maybe 499 Client Closed Request
~f_p_l_socks~
W, [2022-08-12T14:39:33.114649 #24986] WARN -- : Failed to process request to https://www.socks-proxy.net/ (error reading from socket: Could not parse data)
~f_p_l_us~
0 – nothing
~proxy_list~
//8.210.155.X:P
//103.252.1.X:P
~proxy_list_download_http~
0 – nothing
~proxy_list_download_https~
0 – nothing
~proxy_list_download_socks4~
0 – nothing
~proxy_list_download_socks5~
0 – nothing
~proxyscrape_http~
//85.14.243.X:P
//89.208.219.X:P
//121.156.109.X:P
//103.252.1.X:P
//81.23.100.X:P
~proxyscrape_socks4~
0 – nothing
~proxyscrape_socks5~
0 – nothing
~scapingant~
0 – nothing
~xroxy~
hangs/ infinite processing
In TUI-standalone-mode result = 11. In ProxyFetcher::Client, I think, result would be 10 times greater, and at last better result would be with own client implementation around headless chrome (Ferrum(capybara+cuprite / vessel))
.
@JuPlutonic I don't work with it anymore. Feel free to complete MR.
Ok, the PR/MR will be tested after the 0.16.0/main rebase on it and after recheck. App with the ProxyFetcher::Client will be run and logged with providers mentioned above with ISPs: МТС/another.