charm-tools
charm-tools copied to clipboard
juju
'utf8' codec can't decode byte 0xf6 in position 114: invalid start byte
Checklist
- [X] Confirmed this is an issue with charm-tools, not charmstore-client
- [x] Provide versions of tools used
- [x] Described the feature or ways to replicate the issue
What version am I running?
snap info charm
name: charm
summary: charm and charm-tools
publisher: charms
contact: [email protected]
license: unknown
description: |
charmstore-client and charm-tools
commands:
- charm
snap-id: 2Rryoc2ylScfbFl4eQtpntHD9iuZuMvt
tracking: stable
refreshed: 2017-11-02T05:23:28+03:00
installed: 2.2.3 (114) 103MB -
channels:
stable: 2.2.3 (114) 103MB -
candidate: 2.2.3 (114) 103MB -
beta: ↑
edge: 2.2.4-dev+git-12-g72b50bd (141) 100MB -
Issue/Feature
Ran tox -e pep8 in a charm interface directory which resulted in .tox getting created:
➜ charm-interface-keystone-fid-service-provider git:(master) ✗ tree .tox
.tox
├── log
└── pep8
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
...
Then tried building a charm that depends on this interface:
➜ charm-keystone-saml-mellon git:(master) ✗ grep interface src/layer.yaml includes: ['layer:openstack', 'layer:leadership', 'interface:keystone-fid-service-provider', 'interface:websso-fid-service-provider', 'interface:juju-info']
➜ charm-keystone-saml-mellon git:(master) ✗ tree interfaces/ interfaces/ ├── interface-keystone-fid-service-provider -> ../../charm-interface-keystone-fid-service-provider └── interface-websso-fid-service-provider -> ../../charm-interface-websso-fid-service-provider
➜ charm-keystone-saml-mellon git:(master) ✗ grep -A12 '[testenv]' tox.ini [testenv] setenv = VIRTUAL_ENV={envdir} PYTHONHASHSEED=0 TERM=linux LAYER_PATH={toxinidir}/layers INTERFACE_PATH={toxinidir}/interfaces JUJU_REPOSITORY={toxinidir}/build passenv = http_proxy https_proxy install_command = pip install {opts} {packages} deps = -r{toxinidir}/requirements.txt
➜ charm-keystone-saml-mellon git:(master) ✗ grep -A3 'testenv:build' tox.ini [testenv:build] basepython = python2.7 commands = charm-build --log-level DEBUG -o {toxinidir}/build src {posargs}
I expect/expected the following
It appears to be that everything in the interface directory is imported blindly and it may lead to errors which are fairly cryptic such as the one below. Removing a '.tox' directory in the interface directory solves the problem but may take some time to figure out. I am open to suggestions on how to better handle this but it might be worthwhile to maintain an inclusion ignore list for common hidden directories or auxiliary files.
What I got
grep -A5 testenv:build tox.ini
[testenv:build]
basepython = python2.7
commands =
charm-build --log-level DEBUG -o {toxinidir}/build src {posargs}
[testenv:py27]
tox -e build
build installed: asn1crypto==0.24.0,blessings==1.6,cffi==1.11.5,charm-tools==2.2.3,Cheetah==2.4.4,colander==1.0b1,configparser==3.5.0,cryptography==2.2.2,distro==1.2.0,dnspython==1.15.0,ecdsa==0.13,entrypoints==0.2.3,enum34==1.1.6,functools32==3.2.3.post2,httplib2==0.11.3,idna==2.6,ipaddress==1.0.22,Jinja2==2.10,jsonschema==2.5.1,jujubundlelib==0.5.6,keyring==12.2.0,launchpadlib==1.10.6,lazr.restfulclient==0.14.0,lazr.uri==1.0.3,libcharmstore==0.0.7,Markdown==2.6.11,MarkupSafe==1.0,netaddr==0.7.19,netifaces==0.10.7,oauthlib==2.0.7,otherstuf==1.1.0,paramiko==1.18.5,parse==1.8.2,path.py==8.1.2,pathspec==0.3.4,pbr==1.8.1,pkg-resources==0.0.0,psutil==1.2.1,pycparser==2.18,pycrypto==2.6.1,PyYAML==3.12,requests==2.9.1,ruamel.base==1.0.0,ruamel.ordereddict==0.4.13,ruamel.yaml==0.10.23,SecretStorage==2.3.1,simplejson==3.14.0,six==1.11.0,stuf==0.9.16,testresources==2.0.1,theblues==0.3.8,translationstring==1.3,virtualenv==15.2.0,wadllib==1.3.2
build runtests: PYTHONHASHSEED='0'
build runtests: commands[0] | charm-build --log-level DEBUG -o /home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build src
build: Destination charm directory: /home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/builds/keystone-saml-mellon
build: {
"INTERFACE_PATH": "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/interfaces",
"JUJU_REPOSITORY": "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build",
"LAYER_PATH": "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/layers",
"_charm": "src",
"_charm_metadata": {
"description": "The main goal of this charm is to generate the necessary configuration for use in the Keystone charm related to Service Provider config generation, trust establishment between a remote idP and SP via certificates and signaling Keystone service restart. Keystone has a concept of a federated backend which serves multiple purposes including being a backend part of a Service Provider in an authentication scenario where SAML is used. Unless ECP is used on a keystone client side, SAML-related exchange is performed in an Apache authentication module (Mellon in case of this charm) and SAML assertions are converted to WSGI environment variables passed down to a particular mod_wsgi interpreter running Keystone code. Keystone has an authentication plug-in called \"mapped\" which does the rest of the work of resolving symbolic attributes and using them in mappings defined by an operator or validating the existence of referenced IDs.",
"maintainer": "OpenStack Charmers <[email protected]>",
"name": "keystone-saml-mellon",
"provides": {
"keystone-fid-service-provider": {
"interface": "keystone-fid-service-provider",
"scope": "container"
},
"websso-fid-service-provider": {
"interface": "websso-fid-service-provider",
"scope": "global"
}
},
"requires": {
"container": {
"interface": "juju-info",
"scope": "container"
}
},
"resources": {
"idp-metadata": {
"description": "Identity Provider metadata XML file that conforms to\nsaml-metadata-2.0-os specification. This file contains idP\nidentification information and its certificates with public keys\nthat can be used for signing and encryption on the idP side in\nIDPSSODescriptor and other information which can be used on the\nservice provider side to interact with that idP.\n",
"filename": "idp-metadata.xml",
"type": "file"
},
"sp-private-key": {
"description": "Private key used by Service Provider (mod_auth_mellon) to sign\nand/or SAML-level (not transport-level) encryption.\n",
"filename": "sp-private-key.pem",
"type": "file"
},
"sp-signing-keyinfo": {
"description": "Specifies a signing KeyInfo portion of SPSSODescriptor to be used\nin Service Provider metadata. This should be an XML portion\nwhich in the simplest case is formatted as shown below:\nThis fragment should contain a certificate that contains a public\nkey of a Service Provider in case an idP requires that SAML\nrequests are signed.\nThe term \u201csigning certificate\u201d is a misnomer. A signing\ncertificate in metadata is actually used for signature\nverification, not signing. The private signing key is held\nsecurely by the signing party (SP in this case). In a SAML\nexchange an SP signs SAML messages with its private key and idP\nvalidates them via a public key embedded in a certificate present\nin the SP's metadata XML and vice versa for idP.\n",
"filename": "sp-signing-keyinfo.xml",
"type": "file"
}
},
"series": [
"xenial",
"bionic",
"artful",
"trusty"
],
"subordinate": true,
"summary": "Federated identity with SAML via Mellon Service Provider",
"tags": [
"openstack",
"identity",
"federation",
"idP"
]
},
"_name": "keystone-saml-mellon",
"_top_layer": null,
"config": "<charmtools.build.config.BuildConfig object at 0x7ffa33abbf50>",
"deps": "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/deps",
"description": false,
"force": false,
"hide_metrics": false,
"interface_service": "https://juju.github.io/layer-index/",
"log_level": "DEBUG",
"no_local_layers": false,
"output_dir": "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build",
"repo": "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/builds",
"report": false,
"series": null,
"target_dir": "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/builds/keystone-saml-mellon",
"verbose": false,
"wheelhouse_overrides": null
}
build: Please add a `repo` key to your layer.yaml, e.g. repo: [email protected]:dshcherb/charm-keystone-saml-mellon.git
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/layers/openstack.json
charmtools.build.fetchers: Found repo: https://github.com/openstack/charm-layer-openstack
charmtools.fetchers: git clone https://github.com/openstack/charm-layer-openstack /home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/deps/layer/tmp9ls3q7: Cloning into '/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/deps/layer/tmp9ls3q7'...
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/layers/basic.json
charmtools.build.fetchers: Found repo: https://github.com/juju-solutions/layer-basic.git
charmtools.fetchers: git clone https://github.com/juju-solutions/layer-basic.git /home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/deps/layer/tmpgBLMGA: Cloning into '/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/deps/layer/tmpgBLMGA'...
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/layers/leadership.json
charmtools.build.fetchers: Found repo: https://git.launchpad.net/layer-leadership
charmtools.fetchers: git clone https://git.launchpad.net/layer-leadership /home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/deps/layer/tmpbgl60c: Cloning into '/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/deps/layer/tmpbgl60c'...
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/interfaces/juju-info.json
charmtools.build.fetchers: Found repo: https://github.com/juju-solutions/interface-juju-info.git
charmtools.fetchers: git clone https://github.com/juju-solutions/interface-juju-info.git /home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/deps/interface/tmpT7cCQC: Cloning into '/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build/deps/interface/tmpT7cCQC'...
build: Processing layer: layer:basic
build: Processing layer: layer:openstack
build: Processing layer: layer:leadership
build: Processing layer: keystone-saml-mellon (from src)
build: Processing interface: keystone-fid-service-provider (from interfaces/interface-keystone-fid-service-provider)
build: Processing interface: websso-fid-service-provider (from interfaces/interface-websso-fid-service-provider)
build: Processing interface: juju-info
Traceback (most recent call last):
File "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/bin/charm-build", line 11, in <module>
sys.exit(main())
File "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/local/lib/python2.7/site-packages/charmtools/build/builder.py", line 750, in main
build()
File "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/local/lib/python2.7/site-packages/charmtools/build/builder.py", line 560, in __call__
self.generate()
File "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/local/lib/python2.7/site-packages/charmtools/build/builder.py", line 513, in generate
self.exec_plan(self.plan, self.layers)
File "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/local/lib/python2.7/site-packages/charmtools/build/builder.py", line 477, in exec_plan
cont &= tactic.lint()
File "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/local/lib/python2.7/site-packages/charmtools/build/tactics.py", line 304, in lint
from_name=relpath)
File "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/local/lib/python2.7/site-packages/charmtools/utils.py", line 508, in delta_python_dump
for lineno, last, current in delta_python(orig, dest, patterns, context):
File "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/local/lib/python2.7/site-packages/charmtools/utils.py", line 452, in delta_python
od = orig.text()
File "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/local/lib/python2.7/site-packages/path.py", line 796, in text
return U_NEWLINE.sub('\n', f.read())
File "/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/lib/python2.7/codecs.py", line 314, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf6 in position 114: invalid start byte
ERROR: InvocationError: '/home/dima/src/canonical/openstack/charm-keystone-saml-mellon/.tox/build/bin/charm-build --log-level DEBUG -o /home/dima/src/canonical/openstack/charm-keystone-saml-mellon/build src'
As I mentioned when we discussed this, the ignore option in the layer.yaml or interface.yaml with a .tox entry would fix this. What probably (maybe) is needed is something in the documentation about "ensure that any artefacts or files/directories that you DON'T want in the final built charm should be including in an ignore option in the ...yaml file" or something equivalent to that. Maybe in https://jujucharms.com/docs/stable/developer-layer-example.
e.g. in the keystone interface.yaml layer:
name: keystone
summary: Interface for integrating with Keystone identity service
maintainer: OpenStack Charmers <[email protected]>
ignore:
- 'unit_tests'
- 'Makefile'
- '.testr.conf'
- 'test-requirements.txt'
- 'tox.ini'
- '.gitignore'
- '.gitreview'
- '.unit-state.db'
