Missing proc_exit event

[nearapogee]

I have used a different version of this God config file in the past an it worked like a charm. However, now will start unicorn and check memory usage but does not seem to notice when the process exists.

I am using god to monitor several other resque processes on the same server using events and things seem to be working well. I am running Ubuntu 10.04.4 LTS. Linux 2.6.32-38-generic-pae #83-Ubuntu SMP Wed Jan 4 12:11:13 UTC 2012 i686 GNU/Linux

Where am I going wrong? (Config files below.)

$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god -D --log-level debug                                            
[sudo] password for deploy:                                                                                                             
I [2012-03-05 08:07:20]  INFO: Syslog enabled.                                                                                          
I [2012-03-05 08:07:20]  INFO: Using pid file directory: /var/run/god                                                                   
I [2012-03-05 08:07:20]  INFO: Started on drbunix:///tmp/god.17165.sock                                                                 
I [2012-03-05 08:07:32]  INFO: unicorn-master Loaded config                                                                             
I [2012-03-05 08:07:32]  INFO: unicorn-master move 'unmonitored' to 'init'                                                              
D [2012-03-05 08:07:32] DEBUG: driver schedule #<God::Conditions::ProcessRunning:0xa2b9148> in 0 seconds                                
I [2012-03-05 08:07:32]  INFO: unicorn-master moved 'unmonitored' to 'init'                                                             
I [2012-03-05 08:07:32]  INFO: unicorn-master [trigger] process is not running (ProcessRunning)                                         
D [2012-03-05 08:07:32] DEBUG: unicorn-master ProcessRunning [false] {true=>:up, false=>:start}                                         
I [2012-03-05 08:07:32]  INFO: unicorn-master move 'init' to 'start'                                                                    
I [2012-03-05 08:07:32]  INFO: unicorn-master before_start: no pid file to delete (CleanPidFile)                                        
I [2012-03-05 08:07:32]  INFO: unicorn-master start: /home/deploy/.rbenv/versions/1.9.3-p0/bin/unicorn -D -c /home/deploy/apps/project/current/config/unicorn.rb                                                                                                             
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::ProcessRunning:0xa2b8f7c> in 0 seconds                                
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::Tries:0xa2b8f04> in 0 seconds                                         
I [2012-03-05 08:07:35]  INFO: unicorn-master moved 'init' to 'start'                                                                   
I [2012-03-05 08:07:35]  INFO: unicorn-master [trigger] process is running (ProcessRunning)                                             
D [2012-03-05 08:07:35] DEBUG: unicorn-master ProcessRunning [true] {true=>:up}                                                         
I [2012-03-05 08:07:35]  INFO: unicorn-master move 'start' to 'up'                                                                      
I [2012-03-05 08:07:35]  INFO: unicorn-master registered 'proc_exit' event for pid 3155                                                 
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::MemoryUsage:0xa2b89b4> in 0 seconds                                   
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::CpuUsage:0xa2b8860> in 0 seconds                                      
I [2012-03-05 08:07:35]  INFO: unicorn-master moved 'start' to 'up'                                                                     
I [2012-03-05 08:07:35]  INFO: unicorn-master [ok] memory within bounds [7468kb] (MemoryUsage)                                          
D [2012-03-05 08:07:35] DEBUG: unicorn-master MemoryUsage [false] {true=>:restart}                                                      
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::MemoryUsage:0xa2b89b4> in 30 seconds                                  
I [2012-03-05 08:07:35]  INFO: unicorn-master [ok] cpu within bounds [0.0%%] (CpuUsage)                                                 
D [2012-03-05 08:07:35] DEBUG: unicorn-master CpuUsage [false] {true=>:restart}                                                         
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::CpuUsage:0xa2b8860> in 30 seconds 

Then kill 3155

I [2012-03-05 08:16:05]  INFO: unicorn-master [ok] memory within bounds [7476kb, 7476kb, 0kb] (MemoryUsage)
D [2012-03-05 08:16:05] DEBUG: unicorn-master MemoryUsage [false] {true=>:restart}
D [2012-03-05 08:16:05] DEBUG: driver schedule #<God::Conditions::MemoryUsage:0xa2b89b4> in 30 seconds
I [2012-03-05 08:16:05]  INFO: unicorn-master [ok] cpu within bounds [0.0022087244616234127%%, 0.002071465561885034%%, 0%%] (CpuUsage)
D [2012-03-05 08:16:05] DEBUG: unicorn-master CpuUsage [false] {true=>:restart}
D [2012-03-05 08:16:05] DEBUG: driver schedule #<God::Conditions::CpuUsage:0xa2b8860> in 30 seconds

$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god status
unicorn:
  unicorn-master: up

As you can see god still thinks the process is up, even though it shows 0 for memory usage. The pid file does exist in the specified location when the process is running. Below is the out put from -V and check, along with the God and Unicorn config files.

$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god -V
Version: 0.12.1
Polls: enabled
Events: netlink

$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god check
using event system: netlink
starting event handler
forking off new process
forked process with pid = 3227
killing process
[ok] process exit event received

God config file.

God.watch do |w|
  w.name = "unicorn-master"
  w.group = "unicorn"

  w.dir = "/home/deploy/apps/project/current"
  w.interval = 30.seconds
  w.env = { 'RAILS_ENV' => 'production' }
  w.start = "/home/deploy/.rbenv/versions/1.9.3-p0/bin/unicorn -D -c /home/deploy/apps/project/current/config/unicorn.rb"
  w.pid_file = "/home/deploy/apps/project/shared/pids/unicorn.pid"

  w.uid = "deploy"
  w.gid = "deploy"

  w.behavior(:clean_pid_file)

  # With events
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) { |c| c.running = true }
  end
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) { |c| c.running = true }
    on.condition(:tries) { |c| c.times = 5; c.transition = :start }
  end
  w.transition(:up, :start) do |on|
    on.condition(:process_exits)
  end
  w.transition(:up, :restart) do |on|
    on.condition(:memory_usage) { |c| c.above = 24.megabytes; c.times = [2,3] }
    on.condition(:cpu_usage) { |c| c.above = 8.percent; c.times = [2,3] }
  end

  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minutes
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

Unicorn (ver 4.2.0) config file

worker_processes 2
working_directory "/home/deploy/apps/project/current"
listen "/tmp/project.socket", :backlog => 64
timeout 30
user 'deploy', 'deploy'
pid "/home/deploy/apps/project/shared/pids/unicorn.pid"
stderr_path "/home/deploy/apps/project/shared/log/unicorn.stderr.log"
stdout_path "/home/deploy/apps/project/shared/log/unicorn.stdout.log"
before_fork do |server, worker|
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
  old_pid = "#{server.config[:pid]}.oldbin"
  if old_pid != server.pid
    begin
      sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
      Process.kill(sig, File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
    end
  end
  sleep 1 # Prevent signals from being lost when busy
end
after_fork do |server, worker|
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

[scottbartell]

Did you ever figure this out?