whenever-elasticbeanstalk
whenever-elasticbeanstalk copied to clipboard
dignoe
Command cron_01_set_leader failed
After installing/configuring this gem, I'm seeing the following error in /var/log/cfn-init.log on my EC2 instance after running git aws.push from my local repo.
2014-10-21 08:08:37,602 [DEBUG] Running test for command cron_01_set_leader 2014-10-21 08:08:37,744 [DEBUG] Test command output: 2014-10-21 08:08:37,745 [DEBUG] Test for command cron_01_set_leader passed 2014-10-21 08:08:38,085 [ERROR] Command cron_01_set_leader (su -c "/usr/local/bin/bundle exec create_cron_leader --no-update" $EB_CONFIG_APP_USER) failed 2014-10-21 08:08:38,086 [DEBUG] Command cron_01_set_leader output: bash: /usr/local/bin/bundle: No such file or directory
2014-10-21 08:08:38,086 [ERROR] Error encountered during build of postbuild_0_tubett: Command cron_01_set_leader failed Traceback (most recent call last):
I have added the whenever-elasticbeanstalk Below is my cron.config file content.. Any idea ...what am i doing wrong?
files:
Reload the on deployment
/opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh: mode: "00700" owner: root group: root content: | #!/usr/bin/env bash . /opt/elasticbeanstalk/containerfiles/envvars cd $EB_CONFIG_APP_CURRENT su -c "/usr/local/bin/bundle exec setup_cron" $EB_CONFIG_APP_USER
Add Bundle to the PATH
"/etc/profile.d/bundle.sh": mode: "000755" owner: root group: root content: | #!/usr/bin/env bash export PATH=$PATH:/usr/local/bin encoding: plain container_commands: cron_01_set_leader: test: test ! -f /opt/elasticbeanstalk/containerfiles/.cron-setup-complete leader_only: true cwd: /var/app/ondeck command: su -c "/usr/local/bin/bundle exec create_cron_leader --no-update" $EB_CONFIG_APP_USER cron_02_write_cron_setup_complete_file: cwd: /opt/elasticbeanstalk/containerfiles command: touch .cron-setup-complete
Thanks Sumit
Cross post @ http://stackoverflow.com/questions/26483509/command-cron-01-set-leader-output-bash-usr-local-bin-bundle-no-such-file-or
Can you please provide further information on the instance you are attempting to use this with? What is the full path to bundle on your instance? If this has changed, we have an issue obviously.
Seeing the same issue with bundle not existing where the script expects it. I'm running the "64bit Amazon Linux 2014.03 v1.0.9 running Ruby 2.1 (Puma)" instance and bundle exists at "/opt/rubies/ruby-2.1.2/bin/bundle". Why not incorporate something along the lines of BUNDLE_EXE=which bundle to the script and running off of the result?
Digging further, this creates other issues. I'm now seeing this:
2014-11-03 21:00:05,949 [INFO](13482 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Executing script: /opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh
2014-11-03 21:00:06,443 [INFO](13482 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Output from script: bundler: command not found: setup_cron
Install missing gem executables with bundle install
2014-11-03 21:00:06,443 [ERROR](13482 MainThread) [directoryHooksExecutor.py-33] [root directoryHooksExecutor error] Script /opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh failed with returncode 127
@tboyko that looks like a better approach. One of the problems I've found in investigating this on the older instance is the inability to find the gem even if the bundle is correct. Will post some extra info shortly.
Looks like the path issue is throughout the gem. This is tough for me to test (with a forked repo) because my experience with EB is that it just doesn't play nicely with git repos or local paths in the Gemfile...
@tboyko ah yeah… that's a "fun" part of EB eh. I've found ways and means to cache into /vendor (simlinked into /var/app/containerfiles for persistency across deployments )
@sumit20rai and @tboyko can you please try release v1.1.5 on your codebase. Note: you'll need to remove your .ebextensions/cron.config before running wheneverize-eb again.
There seems to be an issue. Here's what I got after upgrading, deleting and recreating cron.config, and deploying:
2014-11-07 17:50:08,242 [DEBUG] Running command a_elasticbeanstalk_support_path 2014-11-07 17:50:08,242 [DEBUG] No test for command a_elasticbeanstalk_support_path 2014-11-07 17:50:08,248 [ERROR] Command a_elasticbeanstalk_support_path (. /tmp/update_appenv_include_eb_config_support.sh) failed 2014-11-07 17:50:08,248 [DEBUG] Command a_elasticbeanstalk_support_path output: containerfiles export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/support /tmp/update_appenv_include_eb_config_support.sh: line 8: /opt/elasticbeanstalk/support/envvars.d/appenv: No such file or directory
2014-11-07 17:50:08,248 [ERROR] Error encountered during build of prebuild_2_[REMOVED]: Command a_elasticbeanstalk_support_path failed
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
changes['commands'] = CommandTool().apply(self._config.commands)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
raise ToolError(u"Command %s failed" % name)
ToolError: Command a_elasticbeanstalk_support_path failed
2014-11-07 17:50:08,249 [ERROR] Unhandled exception during build: Command a_elasticbeanstalk_support_path failed
Traceback (most recent call last):
File "/opt/aws/bin/cfn-init", line 122, in
Crap. Sorry. Hadn't committed one of the files :(
Change .ebextensions/cron.config to:
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# update_appenv_include_eb_config_support.sh
if test -d /opt/elasticbeanstalk/containerfiles; then
echo "containerfiles"
grep -Fwq "export EB_CONFIG_SUPPORT" /opt/elasticbeanstalk/containerfiles/envvars.d/appenv
if [ $? -eq 1 ]; then
echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/containerfiles"
echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/containerfiles" >> /opt/elasticbeanstalk/support/envvars.d/appenv;
fi
elif test -d /opt/elasticbeanstalk/support; then
echo "support"
grep -Fwq "export EB_CONFIG_SUPPORT" /opt/elasticbeanstalk/support/envvars.d/appenv
if [ $? -eq 1 ]; then
echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/support"
echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/support" >> /opt/elasticbeanstalk/support/envvars.d/appenv;
fi
fi
encoding: plain
2014-11-10 17:33:12,126 [DEBUG] Command a_elasticbeanstalk_support_path output: containerfiles
export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/containerfiles
/tmp/update_appenv_include_eb_config_support.sh: line 8: /opt/elasticbeanstalk/support/envvars.d/appenv: No such file or directory
2014-11-10 17:33:12,126 [ERROR] Error encountered during build of prebuild_2_[removed]: Command a_elasticbeanstalk_support_path failed
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
changes['commands'] = CommandTool().apply(self._config.commands)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
raise ToolError(u"Command %s failed" % name)
ToolError: Command a_elasticbeanstalk_support_path failed
2014-11-10 17:33:12,127 [ERROR] Unhandled exception during build: Command a_elasticbeanstalk_support_path failed
Traceback (most recent call last):
File "/opt/aws/bin/cfn-init", line 122, in <module>
worklog.build(detail.metadata, configSets)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 117, in build
Contractor(metadata).build(configSets, self)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 502, in build
self.run_config(config, worklog)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
changes['commands'] = CommandTool().apply(self._config.commands)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
raise ToolError(u"Command %s failed" % name)
ToolError: Command a_elasticbeanstalk_support_path failed
I think there may be bigger issues at hand here. TL;DR - Amazon Linux AMIs are changing their structure and breaking the whole $EB_ variable notion, which this gem depends on. Seems like a major gem restructuring may be involved :/
https://forums.aws.amazon.com/thread.jspa?messageID=583440򎜐
Oh crap. WTF is Amazon doing? Surely this is stuff that should be well documented and notified to users :(
No kidding. Looks like the OpsWorks platform isn’t immune to this sort of thing either:
http://blogs.aws.amazon.com/application-management/post/Tx2G95J53SKI2E0/AWS-OpsWorks-supports-application-environment-variables http://blogs.aws.amazon.com/application-management/post/Tx2G95J53SKI2E0/AWS-OpsWorks-supports-application-environment-variables
Read the first comment…
On Nov 13, 2014, at 2:01 PM, Joel Courtney [email protected] wrote:
Oh crap. WTF is Amazon doing? Surely this is stuff that should be well documented and notified to users :(
— Reply to this email directly or view it on GitHub https://github.com/dignoe/whenever-elasticbeanstalk/issues/18#issuecomment-62974408.
Ok @tboyko I've done some digging.
As root, you have access to the following shell script (which in turn calls a ruby script using a gem not available as ec2-user).
/opt/elasticbeanstalk/bin/get-config
With that you can identify which data you want to access with the following options:
Usage: get-config CATEGORY [OPTIONS]
Categories:
optionsettings environment option settings that affect instance
container container specific configurations
addons addon configurations
environment environment variables
meta EB environment meta-data
Options:
-k, --key KEY Key of specific configurations.
--output OUTPUT_FORMAT Output format. Can be JSON or YAML. Default is JSON.
-n, --namespace NAMESPACE Otion Setting namespace for retrieval. Only applied to Category optionsettings.
-o, --option-name OPTION_NAME Option Setting name for retrieval. Only applied to Category optionsettings.
-a, --add-on ADDON Add-on name. Only applied to Category addons.
-h, --help Help
This itself is obtained from files on the instance, I believe anyway, at /opt/elasticbeanstalk/deploy/configuration/containerconfiguration
So now the question is, do older instances have this available to them. If so not so bad. If not… :(
Has there been any progress or updates to this on Elastic Beanstalk? Are there any alternatives that would be recommended?
@rmahnovetsky I hacked this together to get things functional for my own project. I'm hoping to clean it up at some point: https://github.com/contently/whenever-elasticbeanstalk/pull/1/files
Not specifically about this gem, but here are a couple resources I found recently on the environment variable issue:
- AWS employee shows workaround on 11/14/2014 (but watch out for the typos that I mentioned 2/18/2015): https://forums.aws.amazon.com/thread.jspa?messageID=583183
- Blog article about solving this issue for delayed_job: http://www.dannemanne.com/posts/post-deployment_script_on_elastic_beanstalk_restart_delayed_job
In summary you probably want to do something like this starting from the 2014.09 AMIs:
# Loading environment data
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
EB_APP_CURRENT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
EB_APP_PIDS_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)
Here are some commands to check the AMI version:
cat /etc/system-release
cat /etc/image-id
cat /etc/yum.conf | grep releasever # Beanstalk locks repositories using this line
Is there any update on this? I'm looking to use whenever on ebs. Should I be using this project? Any guidance would be appreciated, thanks.
I ended up creating a worker environment and using the periodic scheduler. http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
I would love to know if there is a solution in the works or something out there to show how to use this gem and eb.
I am also facing something similar and stuck with Unauthorized access error
Above is your schedule file converted to cron syntax; your crontab file was not updated. I am still getting this as the output
I got this completely working with the latest AMI (2015.03). I based my work of the fork from @sanjayginde, but had to make a few more tweaks. I've got a pull request back to his fork open here: contently/whenever-elasticbeanstalk#2. I recommend that he accept my pull request and then open another one from his fork back here to the main repo, because he did 95% of the work.
Has anyone got this working with the latest AMI 2015.09? Seems to be broken again.
https://github.com/dignoe/whenever-elasticbeanstalk/pull/26
For those with issues who still would like this to work, please provide me with some information on whether or not this works for you.
Problem is EB is not checking stack ruby version and complains bundle with this error
Activity execution failed, because: bundler: command not found Install missing gem executables withbundle install(ElasticBeanstalk::ExternalInvocationError) caused by: bundler: command not found Install missing gem executables withbundle install(Executor::NonZeroExitStatus)
need to tell ruby version where gems are installed by adding this line or manage by environment variables.
. /opt/elasticbeanstalk/support/scripts/use-app-ruby.sh
Hope this solves issue.
