algolia-webcrawler icon indicating copy to clipboard operation
algolia-webcrawler copied to clipboard

Published 20 hours ago • verified publisher icon DeuxHuitHuit

Support redirects instead of throwing a "HTTP 302 error"

Open umeshkalia opened this issue 8 years ago • 7 comments

node app.js is generating HTTP 302 error while accessing definition pages of my website e.g. http://www.alleydog.com/glossary/definition.php?term=Eccentricity Other pages are being crawled successfully. Can you please suggest some solution?

umeshkalia avatar Sep 21 '16 16:09 umeshkalia

Right now, redirects are not supported, so I would recommend to update your sitemap and make sure that all listed urls returns 200. But I'll keep this issue open since we should support 302/301 http statuses (but only on the same origin).

nitriques avatar Oct 06 '16 00:10 nitriques

We started to work on supporting redirects: https://github.com/DeuxHuitHuit/algolia-webcrawler/commit/a113ba22687127d06be3bf6a7cf505d157687c2d

nitriques avatar Dec 06 '18 20:12 nitriques

Also added a relevant commit to this problem: a19af58a86

nitriques avatar Jan 23 '19 21:01 nitriques

I would need this feature also, is it already in the latest release on npm? thx Stephan

wortkotze avatar Mar 25 '19 17:03 wortkotze

@wortkotze Sadly, it is still not done. But, it should be pretty easy to add now...

nitriques avatar Jun 12 '19 15:06 nitriques

@nitriques thx I used your commit and added it manually

wortkotze avatar Jun 12 '19 16:06 wortkotze

@wortkotze That's cool! Wanna send a PR !?!?

nitriques avatar Nov 13 '19 04:11 nitriques