crawler
crawler copied to clipboard
Pre/Post request hooks
Would be nice to have something like pre/post request hooks, for instance to detect if request is banned by host, page does not match defined rules, to save some counters etc.