Conditions for child processes

[jbgo]

I needed support for monitoring the memory usage of child processes CPU/memory usage in a hurry, so I created this proof of concept branch: https://github.com/jbgo/god/commit/918bc278e4ca5b8133fe34db06c30ccb93dcb7f0

Is there any interest in refining this approach and getting it merged into god one day?

I see that there has already been interest and attempts at this same problem previously: https://github.com/mojombo/god/pull/30 https://github.com/mojombo/god/issues/7

Here is an example that I'm currently using. Basically, I wanted to get a notification when a child of one of my resque workers exceeded a CPU or memory usage limit, so I can go back and debug the process. I transition to unmonitored because I want to leave the process running, but I don't need to get hundreds of notifications if I'm away from my computer.

w.transition(:up, :unmonitored) do |on|
  on.condition(:child_memory_usage) do |c|
    c.above =  200.megabytes
    c.times = 3
    c.notify = 'developers'
  end

  on.condition(:child_cpu_usage) do |c|
    c.above = 95.percent
    c.times = 3
    c.notify = 'developers'
  end
end

[davidnewhall]

Hello,

I'm trying to use your patch on god, but I get the following error when running god: E [2012-10-10 00:41:09] ERROR: Unhandled exception in ChildCpuUsage condition - (TypeError): wrong argument type Symbol (expected Proc) /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/system/portable_child_poller.rb:37:in sum_ps_output' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/system/portable_child_poller.rb:33:inps_float' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/system/portable_child_poller.rb:19:in percent_cpu' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/conditions/child_cpu_usage.rb:35:intest' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/task.rb:367:in handle_poll' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:64:inhandle_event' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:181:in initialize' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:179:inloop' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:179:in initialize' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:178:innew' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:178:in initialize' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/task.rb:51:innew' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/task.rb:51:in initialize' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/watch.rb:39:ininitialize' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:293:in new' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:293:intask' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:281:in watch' /etc/god/httpd.god:1 /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:639:inload' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:639:in load' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:638:ineach' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:638:in load' /etc/god/god.conf:12 /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:152:inload' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:152:in load_god_file' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:144:inload_config' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:143:in each' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:143:inload_config' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:72:in default_run' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:80:inrun_in_front' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:23:in dispatch' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:8:ininitialize' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/god:124:in new' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/god:124 /usr/bin/god:19:inload' /usr/bin/god:19

Seems like this doesn't work. Any ideas?

-david

[davidnewhall]

The problem is my version of ruby. It's 1.8.5, and this patch needs 1.8.7. I can't upgrade all of my servers, so this patch wont work for me. :(

[arnodirlam]

Sorry for re-opening an old issue, but this is of critical importance for us as well. Numerous components use sub-processes, most recently puma in clustered mode (for zero-downtime deployment) and it's impossible to monitor them using god. Any chance that this could be implemented?

In any case, thanks a lot for making such a great tool!

[harsh-shoto]

I am using puma with cluster mode and god is not able to monitor the child processes. When can we have this feature implemented?