About the multi-process support

[elct9620]

Hi, there

I test yabeda-promtheus with Passenger, but it seems not working. But the local machine is working well with puma.

After I read the source code, I guess it has some problem when running with a multi-process web server.

In this case, I think to add an option for the user to choose use concurrent-ruby in memory or cache the metrics in another server, ex. redis.

I read the source code about the ActionCable one year ago, it also uses Redis (or PostgreSQL) to share some connection information between process/server. So I think add Redis support is a good idea.

[Envek]

Yes, in case of yabeda-prometheus it wraps original prometheus-client which have open issue about multi-process support.

GitLab forked prometheus-client to prometheus-client-mmap which does support multi-process servers via memory mapped files. I believe that they are totally compatible so that it possible to switch yabeda-prometheus over these two. I will investigate that.

We're not faced with this problem because if we need to run multiple Passengers (or Pumas) we do not use their cluster possibilities to launch multiple processes at the same host but run multiple containers with single process in each and let Prometheus to scrape each container individually.

Another option you can consider for now is prometheus_exporter gem by Discourse. It much more mature by now, support multi-process servers, have a lot of metrics but very opinionated for us and supports only Prometheus.

[elct9620]

The prometheus_exporter seems more fit my necessary, I will change to it for now. But I like yabeda's design, hope you can make it better in the future.

[imella]

@Envek Did you manage to make it work on a Passenger running app?

I changed Passenger's config to only allow one process running (--max-pool-size 1 and --min-instances 1) but it didn't work.

Maybe i'm forgetting something.

Great work on yabeda by the way!

[leshik]

@elct9620 @Envek I second for the support of prometheus-exporter, the claim that the

[prometheus_exporter is] built on assumption that various processes (web, jobs, etc) are able to communicate between them on single machine

is simply not true. Containers might be setup to communicate through e.g. overlay network. Furthermore, it's not always convenient to mount /metrics in the main app, sometimes the separate container is preferred. E.g. in my case, the app is deployed to Docker Swarm with order: start-first and failure_action: rollback. It has two parts – the reverse proxy container with static assets inside, and the rails app. App version should be in sync between the two. I use the healthcheck in proxy container to compare the version with the rails container, which has been reported to prometheus_exporter. I'm able to do this because it's the separate container – if /metrics endpoint were mounted in the main app, I would never able to scrape it as it weren't rolled in production yet (because of update_config order).

Would be nice if yabeda-prometheus could send metrics to prometheus_exporter too.

[elct9620]

@leshik

After I compare prometheus_exporter and prometheus-client-mmap, the point I care about is I didn't want to run another process to share the metrics information, because my application is small and more flexible if I mount it on Rails.

In this case, I think prometheus-client-mmap provides the lightweight option for users who didn't want to run another process to handle it.

[Envek]

@elct9620, try to use yabeda-prometheus 0.1.4 with GitLab's gem:

# Gemfile
gem "prometheus-client-mmap"
gem "yabeda-prometheus"

Please note that you need to specify prometheus_multiproc_dir env variable for it to work:

prometheus_multiproc_dir=/tmp

[elct9620]

@Envek

It seems not work.

In my production environment (Passenger) didn't generate any mmap file in /tmp. I guess it is Passenger problem, so I test it on my mac use puma -w 3 to create multiple processes with puma.

But I still got the same result, if I use puma the result is correct, but if I use puma -w 3 it didn't generate any mmap file and I cannot share metrics between process.

[dsalahutdinov]

Hey! support of gitlab client moved to another gem https://github.com/yabeda-rb/yabeda-prometheus-mmap Try it out.

[Envek]

Closing this issue, as official Prometheus Ruby client got pretty good support for metrics aggregation via DirectFileStore (there are caveats however). See https://github.com/prometheus/client_ruby#data-stores