crawly icon indicating copy to clipboard operation
crawly copied to clipboard

Telemetry metrics support

Open Ziinc opened this issue 4 years ago • 2 comments

This PR introduces telemetry support to allow easy monitoring of crawling operations.

Considered success if:

  1. Can monitor scraping-related events (requests made, redirections, response codes, dropped requests, dropped items, etc)
  2. Can monitor spider-related events (requests stored, items stored, items parsed)
  3. Can monitor engine-related events (spider started, spider stopped, worker/manager crashes)

Originally discussed here: https://github.com/oltarasenko/crawly/issues/141#issuecomment-735245579

Ziinc avatar Dec 10 '20 09:12 Ziinc

@Ziinc just in case, how do we render the telemetry results?

oltarasenko avatar Dec 21 '20 18:12 oltarasenko

I think rendering as a separate package like crawly_live_metrics, like how phoenix does it, would be good But instead of a full-fledge dashboard, i was thinking of widgets that can be simply plugged into any existing phoenix app. So then crawly_ui can make use of it too, without the metrics display being overly coupled with the entire crawly_ui application

Ziinc avatar Dec 22 '20 16:12 Ziinc