twitter-scraper-selenium icon indicating copy to clipboard operation
twitter-scraper-selenium copied to clipboard

Proxy

Open Andreito95 opened this issue 1 year ago • 3 comments

Authenticated proxy doesn't load correctly, if i check "whatismyipaddress" on the driver i get my real IP

Andreito95 avatar Jan 17 '23 21:01 Andreito95

What browser it was? Google Chrome Or Mozilla Firefox? Can you please share more details?

shaikhsajid1111 avatar Jan 26 '23 12:01 shaikhsajid1111

it's the same, proxy (with authentication user/pass) doesn't work with both Chrome or Mozilla. I have 10000 rows of python code with implemented Selenium browser, the only way to user proxy with authentication is Chrome and use an extension with a json file like this (you can find code around the web):

pluginfile = 'proxy_auth_plugin.zip' with zipfile.ZipFile(pluginfile, 'w') as zp: zp.writestr("manifest.json", manifest_json) zp.writestr("background.js", background_js) chrome_options.add_extension(pluginfile)

I am trying to find a better way (with both chrome or mozilla, i don't care) that works without exstentions (i would like to disable extensions but i can't because of this one for proxy on chrome). Another thing i can tell you is that it's not possible to save cookies with Chrome with this extension, because it will give problems with proxy authentication, so i am actually using Chrome without cookies and with this extension to use it with proxy with user/pass auth.

I tried your code and i found lot of good and perfect funcitons, but if you can find a solution for the proxy with authentication it will be really helpful to me. Tried both mozilla and chrome with your code and put right string "user:pass@ip:port", browser opens correctly but if you open "whatismyipaddress.com" it will show you the real IP (and not the proxy one), so it doesn't work.

Andreito95 avatar Jan 26 '23 12:01 Andreito95

I think you're right about this. Selenium doesn't have proper functionality when it comes to using an authenticated proxy. Even I encountered the same problem when I once used an authenticated proxy, yeah that problem is available to use some chrome extensions. I think it is the only option even I see is working. If I find some better solution somewhere I would love to implement it here but currently, I don't have one.

I think if your proxy provider provides whitelisting feature then you should consider using it, just whitelist the server although it doesn't really focus on the problem it does gives you an alternative way.

shaikhsajid1111 avatar Jan 26 '23 13:01 shaikhsajid1111