yabeda-prometheus icon indicating copy to clipboard operation
yabeda-prometheus copied to clipboard

Support Yabeda server in a new process

Open etsenake opened this issue 2 years ago β€’ 3 comments

Should fix #21

I've been working on getting yabeda into our app which runs on passenger. I finally have it running, but faced a couple of issues that I've been able to fix manually in our app but wanted to upstream some of those changes Issue 1: Thread gets killed by passenger as a long-running request Issue 2 βœ… : Metrics become pretty bloated when we haven't deployed in a while I imagine this would be a problem on weekends where the app is up for days. The addition of rack deflater shrunk the size of the metrics logged by close to 90% in my testing:

With Rack deflater Without Rack deflater
1355 15918
1538 16851
1582 17784

Depending on your scrape timeout on your prometheus server this could or could not be a problem ours is set pretty low but has worked fine for other exporters we have.

Issue 3[Just a minor annoyance πŸ˜… ]: When running in clustered mode if app isn't preloaded the other processes or threads error out with Errno::EADDRINUSE doesn't kill the app just pollutes the start up logs a bit

Changes

  • add support for turning on rack deflater for the rack app
  • add support for starting the metrics server in a separate process
  • add support for suppressing Errno::EADDRINUSE when running in clustered mode in production apps

Gearing up to go live Monday with these changes and a couple others others in relation to resque on Monday so any feedback would be greatly appreciated.

Shameless plug: because we use resque I wrote a small extension for metrics around activejob for us to get some insight similar to what y’all have done for sidekiq and faktory. Check it out here

etsenake avatar Nov 06 '22 00:11 etsenake

I could also split this up into multiple PRs if that makes it easier to review unsure who to add as a reviewer though @Envek are you the person to review? πŸ˜…

etsenake avatar Nov 06 '22 16:11 etsenake

Also added your yabeda-activejob to the Yabeda plugin list. Thanks for sharing!

Envek avatar Nov 08 '22 04:11 Envek

Awesome! Apologies for taking so long to get back to this was at a work conference for the past couple days getting those changes in, test some things around calling configure! over the next little bit. I'll ping you once it's ready for another

etsenake avatar Nov 10 '22 12:11 etsenake