crawly
crawly copied to clipboard
Telemetry metrics support
This PR introduces telemetry support to allow easy monitoring of crawling operations.
Considered success if:
- Can monitor scraping-related events (requests made, redirections, response codes, dropped requests, dropped items, etc)
- Can monitor spider-related events (requests stored, items stored, items parsed)
- Can monitor engine-related events (spider started, spider stopped, worker/manager crashes)
Originally discussed here: https://github.com/oltarasenko/crawly/issues/141#issuecomment-735245579
@Ziinc just in case, how do we render the telemetry results?
I think rendering as a separate package like crawly_live_metrics, like how phoenix does it, would be good But instead of a full-fledge dashboard, i was thinking of widgets that can be simply plugged into any existing phoenix app. So then crawly_ui can make use of it too, without the metrics display being overly coupled with the entire crawly_ui application