nodestream icon indicating copy to clipboard operation
nodestream copied to clipboard

[REQUEST] Expose Ingestion Statistics as a Prometheus Server

Open zprobst opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe. For perpetually running pipelines in particular, its hard to get a sense of how much work the pipeline is doing against the database. It would be ideal if the user

Describe the solution you'd like Expand the nodestream run command to configure and run a prometheus metrics server:

nodestream run <<pipeline>> --prometheus-server-addr 127.0.0.1:8080

We already collect metrics as part of the PipelineContext api accessible via get_context() in nodestream.pipeline.meta. This means that we can reuse a lot of existing logic for Prometheus. As well as pipeline or application level metrics, it would be ideal to include some process/system level metrics such as memory and cpu usage.

Describe alternatives you've considered Currently there is max memory usage logged as part of the message but that isn't really good enough. the only other alternative is to monkey patch the system, or use custom components for everything.

Additional context n/A

zprobst avatar Jun 28 '23 15:06 zprobst