saltstack
Support for mocking ext_pillars
Hello,
I was wondering if there were any plans to support ext_pillars in kitchen-salt. I have a situation where I have an ext_pillar that is used as an input for another pillar. I haven't been able to find any documentation regarding if this kind of usage is supported or not. The closest I was able to find is PR16 which was closed.
What I want to do is use kitchen-salt to validate that a SLS pillar file has been written correctly and nothing was fat fingered. Errors in the pillar SLS file mean invalid inputs for the states. I think the way to do this is to allow simulated ext_pillars to be populated before the system pillars. This is similar to the how the_ext_pillar_first_ parameter on the salt master works.
If this is already supported, can someone please point me towards the documentation for this?
Summary of the logs: In the files below I have a wireguard-formula that has inputs being provided by the '../pillar/wireguard.sls' pillar. This pillar extracts data from an ext_pillar called wireguard_data. The real ext_pillar calls a database and retrieves the appropriate IP and keys. I already have a unit test for the real wireguard_data ext_pillar so I don't need this here. What I need from kitchen-salt is to be able to crate a mock ext_pillar with an SLS file that is fully populated and made available to subsequent pillars.
.kitchen.yml
<% @version = '3000.6' %>
driver:
name: docker
user_sudo: false
privileged: true
username: root
forward:
- 80
platforms:
- name: ubuntu-18.04
driver_config:
run_command: /lib/systemd/systemd
suites:
- name: salt-kitchen-test
provisioner:
name: salt_solo
sudo: false
salt_install: bootstrap
salt_bootstrap_options: -x python3 -P stable <%= @version %>
log_level: debug
is_file_root: true
state_top_from_file: true
require_chef: false
dependencies:
- name: wireguard
repo: git
source: git://github.com/energytoolbase/wireguard-formula
state_copy_filter:
- .bundle
- .git
- .gitignore
- .kitchen
- .kitchen.yml
- Gemfile
- Gemfile.lock
pillars:
top.sls:
base:
"*":
- wireguard_data
- wireguard
pillars_from_files:
wireguard_data.sls: tests/pillars/wireguard_data.sls
pillars_from_directories:
- ../pillar
verifier:
name: shell
remote_exec: false
command: python3 -m pytest -s -v tests/
tests/pillars/wireguard_data.sls
wireguard_data:
private_key: TEST_PRIVATE_KEY
public_key: TEST_PUBLIC_KEY
wireguard_ip: 10.0.0.1
../pillar/wireguard.sls
# -*- coding: utf-8 -*-
# vim: ft=yaml
---
wireguard:
tofs:
# For testing purposes
source_files:
wireguard-config-file-interface-wg0-config:
- 'wireguard.conf.jinja'
# Background:
interfaces:
wg0:
Interface:
# If this is not set, we will generate the private key
# (together with the public key) and set it via PostUp
PrivateKey: {{ pillar.get('wireguard_data')['private_key'] }}
ListenPort: 40000
Address: {{ pillar.get('wireguard_data')['wireguard_ip'] }}
Peers:
# We name the peers so we can automatically exclude them
# on the node with the same minion ID.
# It's also easy to alter the dicts via salt.pillar.stack.
blah:
PublicKey: PEER_PUBLIC_KEY
AllowedIPs: 10.0.0.0/16
Endpoint: HOSTNAME:40000
Kitchen Logs
...
[DEBUG ] Reading configuration from /tmp/kitchen/etc/salt/minion
[DEBUG ] Using cached minion ID from /tmp/kitchen/etc/salt/minion_id: 5fa263d2a61c
[DEBUG ] Configuration file path: /tmp/kitchen/etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Grains refresh requested. Refreshing grains.
[DEBUG ] Reading configuration from /tmp/kitchen/etc/salt/minion
[DEBUG ] LazyLoaded zfs.is_supported
[DEBUG ] Determining pillar cache
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] compile template: /tmp/kitchen/srv/pillar/top.sls
[DEBUG ] Jinja search path: ['/tmp/kitchen/srv/pillar', '/tmp/kitchen/srv/spm/pillar']
[PROFILE ] Time (in seconds) to render '/tmp/kitchen/srv/pillar/top.sls' using 'jinja' renderer: 0.008135318756103516
[DEBUG ] Rendered data from file: /tmp/kitchen/srv/pillar/top.sls:
---
base:
"*":
- wireguard_data
- wireguard
[DEBUG ] Results of YAML rendering:
OrderedDict([('base', OrderedDict([('*', ['wireguard_data', 'wireguard'])]))])
[PROFILE ] Time (in seconds) to render '/tmp/kitchen/srv/pillar/top.sls' using 'yaml' renderer: 0.0004165172576904297
[DEBUG ] LazyLoaded confirm_top.confirm_top
[DEBUG ] LazyLoaded compound_match.match
[DEBUG ] compound_match: 5fa263d2a61c ? *
[DEBUG ] LazyLoaded glob_match.match
[DEBUG ] compound_match 5fa263d2a61c ? "*" => "True"
[DEBUG ] compile template: /tmp/kitchen/srv/pillar/wireguard_data.sls
[DEBUG ] Jinja search path: ['/tmp/kitchen/srv/pillar', '/tmp/kitchen/srv/spm/pillar']
[PROFILE ] Time (in seconds) to render '/tmp/kitchen/srv/pillar/wireguard_data.sls' using 'jinja' renderer: 0.0014007091522216797
[DEBUG ] Rendered data from file: /tmp/kitchen/srv/pillar/wireguard_data.sls:
wireguard_data:
private_key: TEST_PRIVATE_KEY
public_key: TEST_PUBLIC_KEY
wireguard_ip: 10.0.0.1
[DEBUG ] Results of YAML rendering:
OrderedDict([('wireguard_data', OrderedDict([('private_key', 'TEST_PRIVATE_KEY'), ('public_key', 'TEST_PUBLIC_KEY'), ('wireguard_ip', '10.0.0.0')]))])
[PROFILE ] Time (in seconds) to render '/tmp/kitchen/srv/pillar/wireguard_data.sls' using 'yaml' renderer: 0.0005438327789306641
[DEBUG ] compile template: /tmp/kitchen/srv/pillar/wireguard.sls
[DEBUG ] Jinja search path: ['/tmp/kitchen/srv/pillar', '/tmp/kitchen/srv/spm/pillar']
[ERROR ] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 19, in top-level template code
jinja2.exceptions.UndefinedError: 'None' has no attribute 'private_key'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 169, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 404, in render_jinja_tmpl
buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'None' has no attribute 'private_key'
[CRITICAL] Rendering SLS 'wireguard' failed, render error:
Jinja variable 'None' has no attribute 'private_key'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 19, in top-level template code
jinja2.exceptions.UndefinedError: 'None' has no attribute 'private_key'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 745, in render_pstate
**defaults)
File "/usr/lib/python3/dist-packages/salt/template.py", line 101, in compile_template
ret = render(input_data, saltenv, sls, **render_kwargs)
File "/usr/lib/python3/dist-packages/salt/renderers/jinja.py", line 70, in render
**kws)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 169, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 404, in render_jinja_tmpl
buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'None' has no attribute 'private_key'
[CRITICAL] Pillar render error: Rendering SLS 'wireguard' failed. Please see master log for details.
It's been on my list to dig in to kitchen-salt to try to make it support this as well. We use https://github.com/jgraichen/salt-tower for this same use case (referencing pillars from other pillars) but testing is a little complicated at the moment. I'm also not sure if testing ext_pillars is supported by kitchen-salt yet but from quick searches through the codebase I think probably not.
My goal isn't to test the ext_pillars. Those are python scripts that can be tested with a normal unit test.
I want to test the regular pillars that use data from ext_pillars as an input. What I really want is a mechanism for mocking ext_pillars.
If 'tests/pillars/wireguard_data.sls' is processed first and added to the pillars in the same way an ext_pillar would be, then the regular pillar '../pillar/wireguard.sls' will have access to the data from the 'ext_pillar' and can be tested successfully.
If you already use ext_pillar, you can simply 'push' the same salt/salt-solo configuration to your Kitchen instances. You can overpass kitchen-salt limited directives pushing a salt-master-like configuration file.
We use pillarstack as an ext_pillar, but alone and this is what we use (extract):
# kitchen.yml
salt_minion_config_template: ./test/config/salt-solo.conf
# test/config/salt-solo.conf
# like a master conffile
file_roots:
- /first/path
ext_pillar:
- stack:
- /path/to/stack/conf
