Hookable/eventable crawler behavior

Open kix opened this issue 11 years ago • 1 comments

Maybe this should be opened against the crawler component, but this looks like an application-wide feature.

We could trigger application-wide events during the crawler's lifecycle when a request is initiated, when a URL is discovered, when a response is retrieved, etc. This would allow us to implement:

Clever and clean output handling (pretty much in @everzet's Behat way, though I'm not suggesting to use decorators)
Passing request/response data to pluggable event listeners. Imagine if we're getting a 500 from a Symfony2 app and this is caught by an event handler that sends an email to the developer, with a profiler link attached.
Passing the whole list of URLs discovered during a run to some kind of listener, to make sure that we don't return 404's for pages that are linked from Google search

And I'm sure there's more use cases.

Dec 15 '14 11:12 kix

Yes, having this in crawler is good idea, Thanks @kix! :+1:

Dec 15 '14 11:12 umpirsky