py-linkedin-jobs-scraper
py-linkedin-jobs-scraper copied to clipboard
Enhancement/Feature Request - Rotating Proxies - Pass Cookie Value from code or run-time.
Hi, Spinlud!
First of all thank you for this amazing project.
I would suggest some features, like if it possible we can pass proxy in a parameter in-order for rotating proxy. Second, it would be great if we can pass Cookie Value inside code or run-time rather than setting "Environment Variable". Or What can be done is that we can pass user name and password for login to scrap or use that to extract cookie value. It might solve the cookie expiration issue.
I have built scraper using Selenium, which logins, get the code from gmail using a python package if needed else continue scraping. I was stuck on bypassing "funCaptcha" despite using Anti-Captcha service and Paid Residential Proxies. If you would like, I would love to help you with this. I am not an expert but still I am available. :)
Hi! Yes the problem of using username/password in the scraper is that you need to solve the captcha programmatically, which is a complex thing to do (and not very scalable since captcha type can be changed at any time). 😔