airflow-cookbook
airflow-cookbook copied to clipboard
bahchis
python_runtime error on python3, python_version == '3.5'
I'm trying to upgrade to Airflow 1.10.1 (with Python3) on CentOS 7 and getting an error related to the python_runtime version. I'm using the version numbers from your default and it's still failing.
I realize that this might not be addressed right away, if at all - it looks like this is because of a dependency in poise-python (which was archived without support for Python 3.5 or 3.6 for RHEL6).
# my Python config
default['airflow']['python_runtime'] = '3'
default['airflow']['python_version'] = '3.5'
error stack from chef-client:
* python_runtime[3] action install
* poise_languages_system[python] action install
================================================================================
Error executing action `install` on resource 'poise_languages_system[python]'
================================================================================
PoiseLanguages::Error
---------------------
Package python would install 2.7.5-86.el7, which does not match 3.5. Please set the package_name or package_version provider options.
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:244:in `block (4 levels) in patch_load_current_resource!'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:233:in `block (3 levels) in patch_load_current_resource!'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `tap'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `block (2 levels) in patch_load_current_resource!'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:211:in `block (2 levels) in run_action_hack'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `each'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `block in run_action_hack'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:196:in `run_action_hack'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:102:in `block in action_install'
/var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `instance_eval'
/var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `subcontext_block'
/var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:67:in `notifying_block'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:100:in `action_install'
/var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
/var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:45:in `action_install'
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/mixin.rb
32: poise_languages_system system_package_name do
33: # Otherwise use the default install action.
34: action(:upgrade) if options['package_upgrade']
35: parent new_resource
36: # Don't pass true because we want the default computed behavior for that.
37: dev_package options['dev_package'] unless options['dev_package'] == true
38: dev_package_overrides dev_package_overrides
39: package_version options['package_version'] if options['package_version']
40: version options['version']
41: instance_exec(&block) if block
42: end
43: end
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/mixin.rb:32:in `install_system_packages'
poise_languages_system("python") do
action [:install]
retries 0
retry_delay 2
default_guard_interpreter :default
declared_type :poise_languages_system
cookbook_name "vox-airflow"
parent # Declared in /var/chef/cache/cookbooks/vox-airflow/recipes/packages.rb:1:in `from_file'
python_runtime("3") do
provider PoisePython::PythonProviders::System
action [:install]
retries 0
retry_delay 2
default_guard_interpreter :default
subresources [python_package[apache-airflow], python_package[greenlet], python_package[eventlet], python_package[celery], python_package[cryptography], python_package[sphinx], python_package[sphinx-argparse], python_package[sphinx-rtd-theme], python_package[Sphinx-PyPI-upload], python_package[docker-py], python_package[druid], python_package[boto3], python_package[httplib2], python_package[google-auth-httplib2], python_package[google-api-python-client], python_package[oauth2client], python_package[PyOpenSSL], python_package[hive-thrift-py], python_package[pyhive], python_package[impyla], python_package[unicodecsv], python_package[jaydebeapi], python_package[pymssql], python_package[mysqlclient], python_package[librabbitmq], python_package[psycopg2], python_package[boto], python_package[filechunkio], python_package[pysmbclient], python_package[statsd], python_package[vertica-python], python_package[ldap3], python_package[pykerberos], python_package[thrift_sasl], python_package[snakebite[kerberos]], python_package[bcrypt], python_package[flask-bcrypt], python_package[Flask-OAuthlib], python_package[qds-sdk], python_package[cloudant], python_package[lxml], python_package[nose], python_package[nose-parameterized], python_package[mock], python_package[click], python_package[moto], python_package[freezegun]]
declared_type :python_runtime
cookbook_name "vox-airflow"
recipe_name "packages"
version "3.5"
pip_version "10.0.1"
get_pip_url "https://bootstrap.pypa.io/get-pip.py"
setuptools_version true
virtualenv_version true
wheel_version true
end
version "3.5"
package_name "python"
dev_package "python-devel"
end
System Info:
------------
chef_version=12.22.3
platform=centos
platform_version=7.4.1708
ruby=ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
program_name=chef-client worker: ppid=30126;start=23:31:37;
executable=/opt/chef/bin/chef-client
================================================================================
Error executing action `install` on resource 'python_runtime[3]'
================================================================================
PoiseLanguages::Error
---------------------
poise_languages_system[python] (/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/mixin.rb line 32) had an error: PoiseLanguages::Error: Package python would install 2.7.5-86.el7, which does not match 3.5. Please set the package_name or package_version provider options.
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:244:in `block (4 levels) in patch_load_current_resource!'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:233:in `block (3 levels) in patch_load_current_resource!'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `tap'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `block (2 levels) in patch_load_current_resource!'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:211:in `block (2 levels) in run_action_hack'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `each'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `block in run_action_hack'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:196:in `run_action_hack'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:102:in `block in action_install'
/var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `instance_eval'
/var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `subcontext_block'
/var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:67:in `notifying_block'
/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:100:in `action_install'
/var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
/var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:45:in `action_install'
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/vox-airflow/recipes/packages.rb
1: python_runtime node['airflow']['python_runtime'] do
2: version node['airflow']['python_version']
3: provider :system
4: pip_version node['airflow']['pip_version']
5: end
6:
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/vox-airflow/recipes/packages.rb:1:in `from_file'
python_runtime("3") do
provider PoisePython::PythonProviders::System
action [:install]
retries 0
retry_delay 2
default_guard_interpreter :default
subresources [python_package[apache-airflow], python_package[greenlet], python_package[eventlet], python_package[celery], python_package[cryptography], python_package[sphinx], python_package[sphinx-argparse], python_package[sphinx-rtd-theme], python_package[Sphinx-PyPI-upload], python_package[docker-py], python_package[druid], python_package[boto3], python_package[httplib2], python_package[google-auth-httplib2], python_package[google-api-python-client], python_package[oauth2client], python_package[PyOpenSSL], python_package[hive-thrift-py], python_package[pyhive], python_package[impyla], python_package[unicodecsv], python_package[jaydebeapi], python_package[pymssql], python_package[mysqlclient], python_package[librabbitmq], python_package[psycopg2], python_package[boto], python_package[filechunkio], python_package[pysmbclient], python_package[statsd], python_package[vertica-python], python_package[ldap3], python_package[pykerberos], python_package[thrift_sasl], python_package[snakebite[kerberos]], python_package[bcrypt], python_package[flask-bcrypt], python_package[Flask-OAuthlib], python_package[qds-sdk], python_package[cloudant], python_package[lxml], python_package[nose], python_package[nose-parameterized], python_package[mock], python_package[click], python_package[moto], python_package[freezegun]]
declared_type :python_runtime
cookbook_name "vox-airflow"
recipe_name "packages"
version "3.5"
pip_version "10.0.1"
get_pip_url "https://bootstrap.pypa.io/get-pip.py"
setuptools_version true
virtualenv_version true
wheel_version true
end
System Info:
------------
chef_version=12.22.3
platform=centos
platform_version=7.4.1708
ruby=ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
program_name=chef-client worker: ppid=30126;start=23:31:37;
executable=/opt/chef/bin/chef-client
hmm, I know poise is being deprecated but we definitely use airflow and this recipe with
default['airflow']['python_runtime'] = '3'
default['airflow']['python_version'] = '3.5'
please upgrade pip and try again:
# cant upgrade to pip 19 until this is fixed: https://github.com/poise/poise-python/pull/142
# https://github.com/poise/poise-python/issues/140
default['airflow']['pip_version'] = '18.0'
# can't use https://bootstrap.pypa.io/3.3/get-pip.py because it doesnt support 3.3
# added this revision url because of this issue: https://github.com/pypa/get-pip/issues/37
# added get_pip_url to airflow-cookbook: https://github.com/pypa/get-pip/issues/37
# default['airflow']['get_pip_url'] = 'https://raw.githubusercontent.com/pypa/get-pip/f88ab195ecdf2f0001ed21443e247fb32265cabb/get-pip.py'
default['airflow']['get_pip_url'] = 'https://bootstrap.pypa.io/3.3/get-pip.py'
