nodestream
nodestream copied to clipboard
[REQUEST] Expose Ingestion Statistics as a Prometheus Server
Is your feature request related to a problem? Please describe. For perpetually running pipelines in particular, its hard to get a sense of how much work the pipeline is doing against the database. It would be ideal if the user
Describe the solution you'd like
Expand the nodestream run
command to configure and run a prometheus metrics server:
nodestream run <<pipeline>> --prometheus-server-addr 127.0.0.1:8080
We already collect metrics as part of the PipelineContext
api accessible via get_context()
in nodestream.pipeline.meta
. This means that we can reuse a lot of existing logic for Prometheus. As well as pipeline or application level metrics, it would be ideal to include some process/system level metrics such as memory and cpu usage.
Describe alternatives you've considered Currently there is max memory usage logged as part of the message but that isn't really good enough. the only other alternative is to monkey patch the system, or use custom components for everything.
Additional context n/A