pyspider icon indicating copy to clipboard operation
pyspider copied to clipboard

<base href='relative_path' is not interpreted properly.

Open agarwal-nitesh opened this issue 3 years ago • 0 comments

  • pyspider version: latest version, built the master branch using docker.
  • Operating system: macOS Catalina
  • Start up command: docker-compose -f docker-compose.yaml build && docker-compose up

Expected behavior

Either the response should interpret the base tags or, on setting response.url and response.orig_url, proper hrefs are constructed for relative hrefs.

Actual behavior

Proper href URLs are not formed.

How to reproduce

Try to crawl a website with <base href="...."> tag.

agarwal-nitesh avatar Jan 23 '22 06:01 agarwal-nitesh