mushorg
KeyError: '/status_404'
-> it's giving error if we are cloning website which have error page. becaue
def add_scheme(url): if url[-1] == '/': url = url.strip('/') if yarl.URL(url).scheme: new_url = yarl.URL(url) err_url = yarl.URL(url + '/status_404') else: new_url = yarl.URL('http://' + url) err_url = yarl.URL('http://' + url + '/status_404') return new_url, err_url
in the above function from coner.py we are directly adding error page whithout checking whether the website has error page. kindly, let me know if there is any mistake in my comment.
@MaheswarReddy1194321 sorry for the late reply
I can't reproduce your problem :( But I know we had some problems with a cloner, so it doesn't work stable all the time. You should check you meta.json and try probably to re-clone again
As per my observation, this issue arises in 2 cases:
- The target website does not have an error page.
- Upon visiting a page that does not exist, we are redirected to the homepage.
The website in question (used by @MaheswarReddy1194321) comes under the 2nd case as in.yahoo.com/status_404 redirects to in.yahoo.com/?err=404&err_url=https%3a%2f%2fin.yahoo.com%2fstatus_404.
