itsy

itsy copied to clipboard

It'd be nice to change the location of itsy.log. My main partition is pretty small and would fill up pretty quickly during the run I'm doing right now. Not a...

red-bin

added host-specific queues and now host-wide delay policies can be enforced

1

I have added a default-delay-policy (wait 3 seconds before next request). And now, a ready queue is used to look @ the next set of URLs. Increments are done once...

shriphani

Obeying robots.txt?

3

Any plans for robots.txt compliance?

giorgio79

Delay policies

1

Hi, Right now your crawler polls sites quite aggressively. It would be appropriate to wait for a short while before sending another request to a host.

shriphani

Domain limiting

1

Hey dakrone, you mention Itsy domain limiting capabilities, can you elaborate? In this case, I'd like to e.g. extract only pages->text which have a certain domain pattern. Of course I...

clojens