lazynlp icon indicating copy to clipboard operation
lazynlp copied to clipboard

urllib fails without headers

Open Olamyy opened this issue 5 years ago • 0 comments

Hi, Thanks for this great tool.

I noticed urllib fails with a Forbidden Request error when I call download_page on some links. You can reproduce the error by trying the code below:

import lazynlp
link = "https://punchng.com/"
page = lazynlp.download_page(link, context=None, timeout=None)

This raises a 403 as shown below. Screen Shot 2019-09-16 at 2 09 51 PM

I've attempted to create a PR that adds headers to the request by default.

Olamyy avatar Sep 16 '19 13:09 Olamyy