It should be possible to add ignored URL pattern (regex) to recursive crawls. They should be updateable (add/remove pattern) while the job is running.
PromyLOPh
