Consider move from ES to Prometheus for Metrics collection

[yasiryagi]

I highly recommend moving from Elastic search to Prometheus for collecting storage and distributed nodes metrics. This enable each node to collect , store and service it is own metrics. Introducing the following advantages:

• Making the solution more scalable and flexible.
• Increasing the metrics storage capacity.
• Minimize the risk of loosing the data that ES store in a centralized server.
• Industry standard.
• Allow to utilize a more flexible data collection and alerting
• FREE

[traumschule]

How would you incorporate this with joystream/community-repo#656?

[mnaamani]

I don't think there is a need to move from one to the other. We can have the best of both worlds. For metrics I think using the new open-telemetry standard is the way to go. So to add our custom metrics collection code we should use https://opentelemetry.io/docs/instrumentation/js/

That said we can collect these metrics and store them both in ES APM as well as in a prometheus time series database so we can use the industry standard and still very good tooling from prometheus ecosystem such as Grafana.

See and example of how this can be setup using the open-telemetry collector: https://github.com/mnaamani/storage-production/blob/main/otel-collector-config.yaml