wget2
wget2 copied to clipboard
Unable to pass without errors
My quest continues to get this running using ansible.
It keeps failing the ansible build because the crawl returns an error:
PLAY RECAP **********************************************************************************************************************************************
localhost : ok=1 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
Which is a blocker for the ansible build to complete.
I am using this:
$ wget2 --recursive --level 100 --no-hsts --adjust-extension --span-hosts --content-disposition --trust-server-names --cut-file-get-vars --exclude-directories=/wp-content,/r --domains sub.domain.com --no-parent sub.domain.com
What happens is, it reports a 404 error on some urls because the website has some broken urls.
I want to completely ignore/skip crawling content in /wp-content
and /r
so I can return a crawl without errors.
Or at least not exit with an error code even if it finds 404.
Is this possible?