Setting env-vars (in .rbenv_vars) is not picked up by the app.

[berkes]

Using unicorn. Setting or changing a env-var requires a restart of unicorn. A simple reload (which is the default when using the capistrano setup) will not make these vars available in the app.

I think the https://github.com/intercity/chef-repo/blob/587ad7bb941b5a52db5964387397a749ea7b28df/vendor/cookbooks/rails/recipes/setup.rb#L32 needs an additional notifies :reload, resources(:service => app_name) to restart the app after changing these.

Case at hand:

• I upgraded a rails app to use the "new" secrets.yml. Feeding that secrets.yml with ENV-variables.
• Created these variables in the node config.
• Ran chef solo. .rbenv-variables now has new vars.
• Deployed updated code with capistrano. Cap reloads unicorn. App fails to come up. secret_key_base is missing.
• Doing a full restart fixed the issue.

I spent about 20- minutes debugging before I found that the issue is simple and the app needed a full restart. Mostly because bundle exec rails c -e production did have the correct env-variables and secrets loaded. I'd love to spend some time to avoid others such downtime (and myself, in a few months when I've forgotten about this issue).

An alternative solution would be to do a full restart instead of a gracefull (zero downtime) reload with capistrano always. But IMHO the better solution is to immediately make the new env-variables available after changing or setting them by rebooting the app on changing the settings.

[ghost]

Hi @berkes,

Thank you for opening the issue. Can you please tell what capistrano plugin do you use for reloading unicorn? Is it https://github.com/capistrano-plugins/capistrano-unicorn-nginx ?

[ghost]

I've found out that capistrano plugins restart unicorn differently. For example capistrano-unicorn sends SIGHUP to master unicorn process:

https://github.com/sosedoff/capistrano-unicorn/blob/master/lib/capistrano-unicorn/capistrano_integration.rb#L103-L113

desc 'Reload Unicorn'
task :reload, :roles => unicorn_roles, :except => {:no_release => true} do
  run <<-END
  if #{unicorn_is_running?}; then
    echo "Reloading Unicorn...";
    #{unicorn_send_signal('HUP')};
  else
    #{start_unicorn}
  fi;
  END
end

This is how it's implemented in capistrano-unicorn-nginx using init script:

https://github.com/capistrano-plugins/capistrano-unicorn-nginx/blob/master/lib/generators/capistrano/unicorn_nginx/templates/unicorn_init.erb#L54-L58

restart|reload)
  sig USR2 && echo reloaded OK && exit 0
  echo >&2 "Couldn't reload, starting '$CMD' instead"
  run "$CMD"
  ;;

I've also found some info about signals handling is in the Unicorn's docs. I'll try both approaches later to see if it has any difference related to env vars loading from .rbenv-vars.

[berkes]

Thanks for looking into this.

Unicorn is reloaded with capistrano, using capistrano3-unicorn. The code to reload is:

namespace :deploy do
  task :restart do
    invoke 'unicorn:restart'
  end

  after 'deploy:publishing', 'restart'
end

This, indeed, implies that Unicorn gets a USR2. In unicorn that means, roughly (This is from my recollection, I may not have understood it correct, don't treat below as a reference, just a rough idea):

• A thread is forked, using the new code.
• If that boots correctly:
  • Unicorn serves the next request using this new code. Spawning new forks if required.
  • Unicorn serves existing requests on old forks, still.
  • Once all old forks are finished, it will kill them off.
  • The forked thread now becomes the main thread.
• You are now running new code without ever having been down.

This does cause some strange behaviour. E.g. connections to databases will need to be renewed; old code continues running, pointing to old assets that may not be there, because asset:precompile removed them. And so on. But that is not important for this issue, just to get an idea of how this works.

And it will be running in the same ENV, since it inherits this from the original thread.

Restarting (not the reload explained above, but a full blown stop-start sequence) will, however, kill off the threads (letting it finish nicely) then start a new thread. This will give some downtime; If there's some long running request of say 2 minutes, you might have 2 minutes dowtime untill this thread can be killed and new ones can be started up.

My personal preference would be to:

• Keep the behaviour of capistrano: reload with zero-downtime.
• Have the chef-repo restart the app when it thinks this is absoluty needed. E.g. after changing the env.[1]

That way you don't need to re-deploy after a change in chef-repo and you'll see the changes to your environment immediately similar to most other changes made in chef-repo (e.g. it will reload nginx after a config change).

---

[1] Other things that might have to restart the app/unicorn are:

• Changing ruby version
• Changing the rails-env (production, staging, development)
• Changing the database_info

[ghost]

Just for information, sending USR2 to the master process is not enough, it should be killed right after new master process is started successfully

[catuss-a]

hi, I don't know if it can help but I added this line below in my unicorn config so it loads new env var values :

root="path/to/current/release"
before_exec do |server|
  ENV.update Hash[*File.read("#{root}/.rbenv-vars").split(/=|\n/)]
end

I'm not using chef repo and capistrano-unicorn3 neither. Hope it'll help