warn-scraper icon indicating copy to clipboard operation
warn-scraper copied to clipboard

Published 20 hours ago • verified publisher icon biglocalnews

Add MN scrape

Open Dilcia19 opened this issue 4 years ago • 1 comments

https://mn.gov/deed/programs-services/dislocated-worker/reports/

PDFs

Dilcia19 avatar Oct 15 '20 21:10 Dilcia19

When I try to scrape using utils.get_urls or requests.get(url) or requests.get(url, verify=False), the website demands that I prove that I'm not a bot: this contains the page it displays. The other error I get is "max retries exceeded ... unable to get local issuer certificate".

The urls of each pdf ends with some seemingly random numbers, so getting the urls of each individual pdf seems impossible. Is there a way to bypass this?

Ash1R avatar Nov 25 '22 07:11 Ash1R