delayed_job OSX Hangs using delayed

OSX El Cap Ruby 2.3.1

When trying to run

bin/delayed_job run on OSX the program hangs while daemons is trying to close any IO in the object space.. Once I give up and hit Ctrl-C - we get:

/Users/sblackstone/code/daemons/lib/daemons/daemonize.rb:128:in `close': Interrupt
    from /Users/sblackstone/code/daemons/lib/daemons/daemonize.rb:128:in `block in close_io'
    from /Users/sblackstone/code/daemons/lib/daemons/daemonize.rb:124:in `each_object'

I opened a ticket with daemons https://github.com/thuehlinger/daemons/issues/50 - and added some debugging support trying to find what io object was trying to be closed. See the ticket for all the details but essentially its getting stuck trying to close a fd which is # a pipe:

jack:import_tester sblackstone$ ./bin/delayed_job run
Skipping #<IO:0x007fae328a2b08>
Skipping #<IO:0x007fae328a2ba8>
Skipping #<IO:0x007fae328a2c48>
Attempting #<File:/Users/sblackstone/code/import_tester/log/development.log>
Attempting #<File:/Users/sblackstone/.rvm/gems/ruby-2.3.1/gems/delayed_job-4.1.2/lib/delayed/command.rb (closed)>
Attempting #<IO:fd 20>

The daemons people suspect this might be an issue with delayed_job - so opening a ticket here now..

Aug 24 '16 17:08 sblackstone

Where are you seeing this loop with DJ code?

Aug 24 '16 17:08 albus522

Full backtrace looks like this:

/Users/sblackstone/code/daemons/lib/daemons/daemonize.rb:128:in `close': Interrupt
    from /Users/sblackstone/code/daemons/lib/daemons/daemonize.rb:128:in `block in close_io'
    from /Users/sblackstone/code/daemons/lib/daemons/daemonize.rb:124:in `each_object'
    from /Users/sblackstone/code/daemons/lib/daemons/daemonize.rb:124:in `close_io'
    from /Users/sblackstone/code/daemons/lib/daemons/daemonize.rb:30:in `simulate'
    from /Users/sblackstone/code/daemons/lib/daemons/application.rb:273:in `start_proc'
    from /Users/sblackstone/code/daemons/lib/daemons/application.rb:296:in `start'
    from /Users/sblackstone/code/daemons/lib/daemons/controller.rb:59:in `run'
    from /Users/sblackstone/code/daemons/lib/daemons.rb:197:in `block in run_proc'
    from /Users/sblackstone/code/daemons/lib/daemons/cmdline.rb:92:in `catch_exceptions'
    from /Users/sblackstone/code/daemons/lib/daemons.rb:196:in `run_proc'
    from /Users/sblackstone/.rvm/gems/ruby-2.3.1/gems/delayed_job-4.1.2/lib/delayed/command.rb:118:in `run_process'
    from /Users/sblackstone/.rvm/gems/ruby-2.3.1/gems/delayed_job-4.1.2/lib/delayed/command.rb:99:in `block in daemonize'
    from /Users/sblackstone/.rvm/gems/ruby-2.3.1/gems/delayed_job-4.1.2/lib/delayed/command.rb:97:in `times'
    from /Users/sblackstone/.rvm/gems/ruby-2.3.1/gems/delayed_job-4.1.2/lib/delayed/command.rb:97:in `daemonize'
    from ./bin/delayed_job:5:in `<main>'

I'll see if I can come up with a minimal example..

Aug 24 '16 17:08 sblackstone

So it is fully within daemons run_proc setup. I hate to play the blame game but I don't see how DJ is the problem here.

Aug 24 '16 17:08 albus522

You might try rolling back to older versions of the daemons.

Aug 24 '16 17:08 albus522

Another thought. Your example is using "run" why is daemons doing anything to a foreground process?

Aug 24 '16 17:08 albus522

albus522 - Docker requires the process to run in foreground - if the program run exits, it the container exits...

So I was trying to just use the same methodology as I do in production

I'm going to try to see what I can figure out on this - will get back to this ticket when I have more.

Aug 24 '16 17:08 sblackstone

Alright, so it only occurs when rails is running in development mode. If you run bin/delayed_job run with daemons, it will hang..

But then if you killall -9 fsevent_watch (part of Rails dev mode) - it will then get past the hung IO.close and enter in into the normal run loop and start processing jobs..

Aug 24 '16 18:08 sblackstone

I wasn't saying you shouldn't use run. I was saying when run in the foreground daemons shouldn't need to do anything to the IO streams. Resetting the IO is part of how you need to detach the process from the foreground.

Aug 24 '16 18:08 albus522

Ahh, gotcha..

So Calling Daemons.run_proc from within command.rb with "run" causes daemons to call "simulate" which calls close_io

close_io isn't getting along with fsevent_watch and hence it hangs..

I'm not sure who's side this falls to but its a bug.. Perhaps daemons needs a command line option to prevent this behavior?

Aug 24 '16 18:08 sblackstone

This issue still remains a problem for running delayed_job as a containerized process (docker) where I need it to run in the foreground.

Everything is fine in production, its development mode that won't work..

Jan 16 '17 17:01 sblackstone

I ran into this issue as well trying to run in development mode for testing. Using essentially the same workaround you posted a PR for, but it'd be nice to get a fix for this into a delayed_job or daemon release.