algolia-webcrawler

algolia-webcrawler copied to clipboard

Support redirects instead of throwing a "HTTP 302 error"

7

node app.js is generating HTTP 302 error while accessing definition pages of my website e.g. http://www.alleydog.com/glossary/definition.php?term=Eccentricity Other pages are being crawled successfully. Can you please suggest some solution?

umeshkalia

PDF-Crawling

3

Hi, any ideas/plans about crawling Documents, especially PDFs? Regards from germany

kernpunkt-thermann

Does the webcrawler take the `lastmod` node into account?

1

Does the webcrawler take the `` node or already existing URLs present in the Index into account when running? **Current problem**: We have a website with +2000 URLs that that...

mrtnvh

This will make it easier to use!

nitriques