pyspider
pyspider copied to clipboard
<base href='relative_path' is not interpreted properly.
- pyspider version: latest version, built the master branch using docker.
- Operating system: macOS Catalina
- Start up command: docker-compose -f docker-compose.yaml build && docker-compose up
Expected behavior
Either the response
should interpret the base tags or, on setting response.url
and response.orig_url
, proper hrefs are constructed for relative hrefs.
Actual behavior
Proper href URLs are not formed.
How to reproduce
Try to crawl a website with <base href="....">
tag.