estuary icon indicating copy to clipboard operation
estuary copied to clipboard

Autoretrieve Stats - Data Ingestion

Open jcace opened this issue 2 years ago • 2 comments

Modify Autoretrieve to collect an event stream of all retrievals, and write it to our retrieval event database.

This should be possible by implementing a Lassie RetrievalSubscriber in Autoretrieve, and then wiring it up so that every time a retrieval event fires we will save the data to the database.

This will depend on getting our database/schema figured out ref: https://github.com/application-research/estuary/issues/873

jcace avatar Jan 13 '23 19:01 jcace

Updated task description:

  1. Add an endpoint to estuary-metrics to be POST'ed to with retrieval stats. It will follow the struct defined here: https://github.dev/filecoin-project/lassie/blob/main/pkg/eventrecorder/eventrecorder.go#L63

  2. Configure autoretrieve to use Lassie Event Publisher to publish events to estuary metrics API (it is configured by this line here) https://github.dev/application-research/autoretrieve/blob/master/autoretrieve.go#L197

jcace avatar Jan 17 '23 19:01 jcace

estuary-metrics task: https://github.com/application-research/estuary-metrics/issues/18

jcace avatar Jan 17 '23 19:01 jcace