pytest-testinfra
pytest-testinfra copied to clipboard
documentation: passing dynamic role variables to tests when runnning molecule verify
I know this could be batted away as something that is related to testinfra only, however, I am finding that I would like to ask here - would you accept a documentation PR that outlines a strategy for including role variables that you passed to your converge playbook, into your unit tests?
Here's what I am currently doing:
# molecule/default/playbook.yml
---
- name: Converge
hosts: all
vars_files:
- testvars.yml
roles:
- role: my-role
foobar: "{{ test_foobar }}"
Then in a molecule/default/testvars.yml
, I have:
---
test_foobar: barfoo
Then, in my molecule/default/tests/conftest.py
, I do the following:
import os
import pytest
from testinfra.utils.ansible_runner import AnsibleRunner
DEFAULT_HOST = 'all'
inventory = os.environ['MOLECULE_INVENTORY_FILE']
runner = AnsibleRunner(inventory)
runner.get_hosts(DEFAULT_HOST)
@pytest.fixture
def testvars(host):
variables = runner.run(
DEFAULT_HOST,
'include_vars',
'testvars.yml'
)
return variables['ansible_facts']
And in my tests, I can then:
def test_something(host, testvars):
print(testvars['test_foobar'])
As far as I can see, this is the cleanest way (I've digged around in a lot of tickets) of matching up the dynamic variables that you pass to your role and your testinfra pytest tests.
If it isn't, please someone tell me :)
In any case, the main question remains - is there a place we can start to document this on the RTD setup?
Woops, meant to open this on https://github.com/metacloud/molecule, closing! :)
Following https://github.com/metacloud/molecule/issues/1396#issuecomment-406719205, I'm not getting anywhere on the molecule repository. Could you provide any guidance here or could we suggest some sort of an API for this? I'd be happy to implement something.
A simple use case to focus the discussion:
I pass in a variable to my role called
http_port
which I open on the firewall withufw
I test that the vaule ofhttp_port
passed into the role was opened byufw
How do I access thishttp_port
value in my test infra tests?
Testinfra should load host and group variables automatically. What do you mean by "I pass a variable to my role ?", is this a role "default" variable ? I think you can load such variables by calling the proper api from ansible (see https://github.com/philpep/testinfra/blob/master/testinfra/utils/ansible_runner.py)
What do you mean by "I pass a variable to my role ?", is this a role "default" variable ?
When I do this:
# molecule/default/playbook.yml
---
- name: Converge
hosts: all
vars_files:
- testvars.yml
roles:
- role: my-role
foobar: "{{ test_foobar }}"
So, here, I meant that I pass a value to foobar. I want to be able to access foobar in my test.
I think you can load such variables by calling the proper api from ansible
I haven't been able to see any value for foobar when I use runner.get_variables()
, should I?
I've seen you mention that testinfra can't know about my playbooks but I assumed that it could pick up variables that are passed into the role. Hope that makes sense.
I had a similar problem, because I wanted to access the role defaults and vars. I have the following in my conftest.py. I did it this way to keep the variable precedence of ansible intact. I used the cache because I had the problem that the host fixture was teared down and setup for every test which made this really slow.
import pytest
cache = {}
@pytest.fixture(scope='module')
def ansible_vars(request, host):
role_name = request.cls.__name__.replace("Test", "").lower()
if (id(host), role_name) in cache.keys():
return cache[(id(host), role_name)]
ansible_vars = host.ansible("include_vars", "file=roles/%s/defaults/main.yml name=role_defaults" %
(role_name))["ansible_facts"]["role_defaults"]
ansible_vars.update(host.ansible.get_variables())
ansible_vars.update(host.ansible(
"include_vars", "file=roles/%s/vars/main.yml name=role_vars" % (role_name))["ansible_facts"]["role_vars"])
del ansible_vars['role_defaults']
cache[(id(host), role_name)] = ansible_vars
return ansible_vars
Seems like we had pretty much the same idea :)
Thanks for weighing in @barnabasJ! I am sure (from counting tickets on molecule and testinfra repositories) that this is a use case that many people need solved. I must say, your solution does look quite nice!
Perhaps this could be solved as a family of test fixtures that come packaged up.
Any thoughts @philpep?
@barnabasJ : how do you use testinfra? It seems you don't use molecule. In my case, putting a conftest.py file in my test dir doesn't affect my tests. Moreover, I don't understand how you use your functions ansible_vars, more specifically I don't understand what is the request parameter. Could you be more explicit please?
In my current environment it's hard to use docker or vms on my local machine, therefore I don't use molecule. But I write the tests to check if the servers are in the desired state after running ansible-playbooks. I just do it without molecule as a dev tool.
Testinfra
is developed as a pytest plugin which allows us to use all the pytest
features while writing our tests. When you write the tests you just specify host as an argument and it gets injected during runtime. This is because host is a pytest.fixture.
The conftest.py file is used to create fixtures that can be shared between different test files. While creating fixtures it's possible to inject other fixtures, which you can then use to get the functionality you want. request is another fixture built into pytest
that allows to inspect the context from which the fixture is called.
OK thanks for the explaination. I was able to adapt your code to an usage with molecule. I just added the following code to my tests (it works like yours):
import pytest
@pytest.fixture
def get_vars(host):
defaults_files = "file=../../defaults/main.yml name=role_defaults"
vars_files = "file=../../vars/main.yml name=role_vars"
ansible_vars = host.ansible(
"include_vars",
defaults_files)["ansible_facts"]["role_defaults"]
ansible_vars.update(host.ansible(
"include_vars",
vars_files)["ansible_facts"]["role_vars"])
return ansible_vars
Of course it could be very useful if this was integrated natively in Testinfra
, but it could be not so easy because of the many use cases that users could encounter.
i'd definitely love to see something like this officially included, having only used molecule and testinfra for about a week it was one of my biggest gripes.
@dangoncalves that fixtures works great, thanks!
There probably isn't one solution for everything. But if we had a documented example, I think most would be able to adapt it to their use-case.
Question: are there any downsides to this approach of using include_vars
? If you see links above, the molecule core devs seem to think it can lead to false positives. I haven't seen any limitations in my work so far.
Otherwise, can we come to some agreement on adding variable fixtures to the https://github.com/philpep/testinfra/blob/master/testinfra/plugin.py? It appears since these paths are totally configurable, we'll need defaults (like ../../defaults/main.yml
) and then allow to override (using some tricks from https://docs.pytest.org/en/latest/example/parametrize.html).
I'm still not sure I fully grasp the notion of why you need to pass variables around like that and why you need them available for your integration tests? I've read this issue multiple times and I cannot come up with a good reason to do this without it being overly complex and confusing for the person maintaining such a library. I think I would need to see an example repository with these additions included to see why it might be worth while effort to make a recommendation.
I know the Ansible community does not believe unit testing but I feel that this scenario, testing variable combinations in custom roles to be more a unit test not an integration test. Testinfra is a integration test library and I could see adding such a new fixture would cause the test library to behave "weirdly" or "falsey" at times if the user has mis-configured their environment or has files laying around.
If I understand your problem scenario then one of the ways I've solved something like this is to just create a new molecule scenario
and add those custom variables in the converge.yml
playbook. Yes, this will increase the time it takes to test your roles, but it's a small price for not testing or having have to many test fixtures floating around and causing you more grief down the road. Here's the example repo
Anyway, I really like this discussion and curious to learn more about how other folks are doing things.
I'm still not sure I fully grasp the notion of why you need to pass variables around like that
I have laid out a clear example in https://github.com/philpep/testinfra/issues/345#issuecomment-406808758:
A simple use case to focus the discussion:
I pass in a variable to my role called http_port which I open on the firewall with ufw I test that the vaule of http_port passed into the role was opened by ufw How do I access this http_port value in my test infra tests?
I could see adding such a new fixture would cause the test library to behave "weirdly" or "falsey" at times if the user has mis-configured their environment or has files laying around.
You're right, we need to avoid this. The default should always work.
If I understand your problem scenario then one of the ways I've solved something like this is to just create a new molecule scenario and add those custom variables in the converge.yml playbook. Yes, this will increase the time it takes to test your roles, but it's a small price for not testing or having have to many test fixtures floating around and causing you more grief down the road. Here's the example repo
Looking at https://github.com/codylane/ansible-role-pyenv/blob/master/molecule/debian/playbook.yml, you have specified - 2.6.9, - 2.7.15 - 3.6.6
as your pyenv versions you want to install. In your https://github.com/codylane/ansible-role-pyenv/blob/517cd3ab912a2c2cf0c62da485e7976cf55555b2/molecule/debian/tests/test_default.py#L114, you hard code which versions you expect to be installed. In fact, (for whatever reason), you're not testing that the version 3.6.6 directory is present.
This example is proving (IMHO) why we need to be able to access role variables in the tests.
If you could access the pyenv versions values you passed to the role in the test, then you could simply iterate over them and make sure the directories are present. This would mean that your tests stay synced with your role variables.
In my case we would also like to be able to run the test periodically on the servers after provisioning them. But not all the servers are exactly the same. This means hard coding the tests is not always possible. If I use the default_vars during development of the ansible roles. I can just specify the vars in the group_vars/host_vars later the way I would do it if I ran a playbook with multiple roles on multiple servers and the defaults are overwritten and the tests are specific to the servers they're run on.
Ahh, I see both your points now. Thank you for taking the time to help clarify.
lwm - I will fix that missing test, thank you for pointing it out. It was there, then I did a refactor due to travis "timeout" problems and I must have forgot to add it back in.
I've really enjoyed this discussion. I'd like to see if I can take a stab at what I think is being proposed here.
Requirement
When using testinfra to test our ansible roles, we would like to have a way to pass in a group of variables that work for the following cases:
- These variables are passed to our role when we run
pytest
which configures our play. - These variables are available to our system under test in our
test_*.py
where they can also be used to avoid hard coding them in our tests?
I could also see a need where the scope needs to also be a requirement to avoid the "setup" and "teardown" routines. Following the testinfra paradigm there are "module" and "function" scope test fixtures so we would probably want a way to do both for maximum flexibility.
Question
- I've not tried this but can we not use
group_vars
orhost_vars
via the ansible_runner?
Potential Solution Discussion
I wonder if we could explore the following scenarios?
1.) See if we can get ansible_runner to use a dynamic inventory where the dynamic inventory is a JSON blob of configuration. (I'll provide an example of this when I have more time)
2.) I also wonder if it is possible to use group_vars
or host_vars
which should be available to both ansible and testinfra?
3.) I wonder if we create a pytest fixture to push custom facts to the SUT (system under test) which would be available to both ansible and testinfra? I can also provide an example of this if anyone is interested.
Just a quick comment on group_vars/host_vars. Yes the ansible_runner picks up on them but when creating roles I don't use those, I see them as a higher level tool used with playbooks. I feel that roles should be runable/testable with just the defaults in most cases and I use group/host_var to customize the playbook runs for the different target servers.
What do you think?
Thanks! I wonder, as another idea, if we can't get this under https://testinfra.readthedocs.io/en/latest/modules.html#testinfra.modules.ansible.Ansible.get_variables since that is the existing API. That leads into https://github.com/philpep/testinfra/blob/master/testinfra/utils/ansible_runner.py which does use the official Ansible module - perhaps there is something we can do there. I will take a look when I get time.
barnabasJ - I hear you and that makes a lot of sense from a testing perspective. My first thought then is to use a dynamic inventory script (i'm still working on an example) that is configured via a JSOB blob where each group would be a different test scenario of configuration data which is then passed to ansible.
However, the challenge is trying to get at those variables through the 'ansible' library API and it's been a few years since I got intimate with it. It will take a few days as I have free time to sort it out but I've got a few more ideas to try.
lwm - Yup, in my initial testing get_variables
is close but it's missing info that I know we can get at. I just need to massage it a bit to get what we want because hostvars
will give us pretty much everything you both are after.
From initial testing, I think I understand why we cannot see the role variables it's because the ansible_runner does not see a playbook, instead it wraps what a playbook might look like via dictionary object
So, with that said, we will need to introduce some new features for the ansible_runner
to read to the following:
- It should support reading a playbook YAML file from the a directory inside the project root.
- There may be multiple playbook files that need to be read for scenario testing where each environment is different and or otherwise isolated.
- The variables contained within that playbook should be returned via the
get_variables()
routine of the ansible_runner so that the system under test (SUT) will be able to interrogate and or simplify our tests from having hard coded tests.
The result that should be returned is a new dictionary that represents hostvars
+ facts
+ inventory vars
, defaults
, or any other variable passed to a playbook run and it should go out of scope depending on the function scope being tested i.e. (module, session, function).
However, before we introduce this new feature, I'd like to get consensus from the project owner @philpep if this solution would be acceptable to implement?
@lwm and @barnabasJ - Please also chime in if I've missed anything.
Great.
There may be multiple playbook files that need to be read for scenario testing where each environment is different and or otherwise isolated.
Should we use some naming convention to allow picking up and reading of vars files in the scenario folder?
However, before we introduce this new feature, I'd like to get consensus from the project owner
Good idea.
The runner is attached to the host which is module
scoped. Which should be good for most situations, I'm not sure about how this would effect roles developed with molecule, but the variables for a host should probably not change during a test run.
The playbook dictionary object you linked to, is located in the run function which is executed everytime something is run on the target. Parsing the playbook and variables every time a command is invoked is very expensive time-wise. If you really need to reread the variable/playbook files this should have to be called explicitly, because of the performance issues. I would prefer to parse the necessary files during e.g. backend creation, so it only happens once per host.
I'm also not sure if we even need such a complicated setup. Are you guys trying to read variables from the playbook itself too?
Right now it's possible to specify the host from within a module using the
testinfra_host
variable like:
testinfra_hosts = ["ansible://all?ansible_inventory=hosts.ini"]
Maybe it'would be enough to add another option here, like specifying a role dir with the defaults and vars to parse. But like you said it's probably best to wait and see what @philpep thinks.
I did play around with it a little bit. Here is a minimal example. All the needed stuff is already in the ansible_runner
except the playbook.
import ansible.cli.playbook
import ansible.playbook
import pprint
pp = pprint.PrettyPrinter()
cli = ansible.cli.playbook.PlaybookCLI(None)
cli.options = cli.base_parser(
connect_opts=True,
meta_opts=True,
runas_opts=True,
subset_opts=True,
check_opts=True,
inventory_opts=True,
runtask_opts=True,
vault_opts=True,
fork_opts=True,
module_opts=True,
).parse_args([])[0]
cli.normalize_become_options()
cli.options.connection = "smart"
cli.options.inventory = 'hosts.ini'
loader, inventory, variable_manager = (
cli._play_prereqs(cli.options))
playbook = ansible.playbook.Playbook.load('playbook.yml', variable_manager, loader)
pp.pprint(variable_manager.get_vars(playbook.get_plays()[0]))
I just copied most of this from the AnsibleRunnerV2(AnsibleRunnerBase):.__init__(self, host_list):
method. I just added the last 2 lines.
This is just thrown together and thought of as a proof of concept.
@barnabasJ - Thanks for posting that little snippet that is exactly what I was thinking, however, I wonder in your example if are also experiencing only seeing the role default vars?
For example, given playbook.yml
---
- hosts: all
become: true
roles:
- role: codylane.pyenv
pyenv_install_these_pythons:
- 2.7.15
pyenv_user: vagrant
pyenv_group: vagrant
pyenv_root: /home/vagrant/.pyenv
ansible.cfg
[ssh_connection]
pipelining = True
[defaults]
# vault_password_file = .vaultpw
inventory= ./inventory
error_on_missing_handler = True
ansible_managed = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
deprecation_warnings = True
display_skipped_hosts = True
host_key_checking = False
gathering = smart
# gather_subset = all
fact_caching = jsonfile
fact_caching_connection = tmp/ansible-facts
fact_caching_timeout = 1500
roles_path = roles
[diff]
always = yes
context = 10
requirements.yml
---
- src: https://github.com/codylane/ansible-role-pyenv
version: master
name: codylane.pyenv
ansible-galaxy install -r requirements.yml
cat foo.py
#!/usr/bin/env python
# flake8: noqa
import ansible.cli.playbook
import ansible.playbook
import pprint
pp = pprint.PrettyPrinter()
cli = ansible.cli.playbook.PlaybookCLI(None)
cli.options = cli.base_parser(
connect_opts=True,
meta_opts=True,
runas_opts=True,
subset_opts=True,
check_opts=True,
inventory_opts=True,
runtask_opts=True,
vault_opts=True,
fork_opts=True,
module_opts=True,
).parse_args([])[0]
cli.normalize_become_options()
cli.options.connection = "smart"
cli.options.inventory = 'inventory'
loader, inventory, variable_manager = (
cli._play_prereqs(cli.options))
playbook = ansible.playbook.Playbook.load('playbook.yml', variable_manager, loader)
pp.pprint(variable_manager.get_vars(playbook.get_plays()[0]))
when I run python ./foo.py
{'ansible_check_mode': False,
'ansible_diff_mode': True,
'ansible_forks': 5,
'ansible_inventory_sources': 'inventory',
'ansible_play_batch': [u'foo-centos7', u'foo-ubuntu1404'],
'ansible_play_hosts': [u'foo-centos7', u'foo-ubuntu1404'],
'ansible_play_hosts_all': [u'foo-centos7', u'foo-ubuntu1404'],
'ansible_playbook_python': '/Users/codylane/.pyenv/versions/testinfra-issue-345-2.7.15/bin/python',
'ansible_run_tags': [],
'ansible_skip_tags': [],
'ansible_version': {'full': '2.5.5',
'major': 2,
'minor': 5,
'revision': 5,
'string': '2.5.5'},
'groups': {'all': [u'foo-centos7', u'foo-ubuntu1404'],
u'suts': [u'foo-centos7', u'foo-ubuntu1404'],
'ungrouped': []},
'omit': '__omit_place_holder__b70745f194898f5bfbe5e4900d6bb515963e148c',
'play_hosts': [u'foo-centos7', u'foo-ubuntu1404'],
'playbook_dir': u'/Users/codylane/prj/python/testinfra-issue-gists/issue-345',
u'pyenv_activated_plugins': [{u'name': u'pyenv-virtualenv',
u'remote': u'origin',
u'repo': u'https://github.com/pyenv/pyenv-virtualenv',
u'update': True,
u'version': u'master'}],
u'pyenv_git': {u'remote': u'origin',
u'repo': u'https://github.com/pyenv/pyenv.git',
u'update': True,
u'version': u'master'},
u'pyenv_group': u'root',
u'pyenv_install_these_pythons': [u'2.6.9',
u'2.7.15',
u'3.4.8',
u'3.5.5',
u'3.6.6',
u'3.7.0'],
u'pyenv_profiled_script': u'/etc/profile.d/pyenv.sh',
u'pyenv_root': u'/opt/pyenv',
u'pyenv_user': u'root',
'role_names': [u'codylane.pyenv'],
'vars': {'ansible_check_mode': False,
'ansible_diff_mode': True,
'ansible_forks': 5,
'ansible_inventory_sources': 'inventory',
'ansible_play_batch': [u'foo-centos7', u'foo-ubuntu1404'],
'ansible_play_hosts': [u'foo-centos7', u'foo-ubuntu1404'],
'ansible_play_hosts_all': [u'foo-centos7', u'foo-ubuntu1404'],
'ansible_playbook_python': '/Users/codylane/.pyenv/versions/testinfra-issue-345-2.7.15/bin/python',
'ansible_run_tags': [],
'ansible_skip_tags': [],
'ansible_version': {'full': '2.5.5',
'major': 2,
'minor': 5,
'revision': 5,
'string': '2.5.5'},
'groups': {'all': [u'foo-centos7', u'foo-ubuntu1404'],
u'suts': [u'foo-centos7', u'foo-ubuntu1404'],
'ungrouped': []},
'omit': '__omit_place_holder__b70745f194898f5bfbe5e4900d6bb515963e148c',
'play_hosts': [u'foo-centos7', u'foo-ubuntu1404'],
'playbook_dir': u'/Users/codylane/prj/python/testinfra-issue-gists/issue-345',
u'pyenv_activated_plugins': [{u'name': u'pyenv-virtualenv',
u'remote': u'origin',
u'repo': u'https://github.com/pyenv/pyenv-virtualenv',
u'update': True,
u'version': u'master'}],
u'pyenv_git': {u'remote': u'origin',
u'repo': u'https://github.com/pyenv/pyenv.git',
u'update': True,
u'version': u'master'},
u'pyenv_group': u'root',
u'pyenv_install_these_pythons': [u'2.6.9',
u'2.7.15',
u'3.4.8',
u'3.5.5',
u'3.6.6',
u'3.7.0'],
u'pyenv_profiled_script': u'/etc/profile.d/pyenv.sh',
u'pyenv_root': u'/opt/pyenv',
u'pyenv_user': u'root',
'role_names': [u'codylane.pyenv']}}
Which the above isn't quiet right, it's only returning the default vars not the vars overriden in my playbook.yml. I'll tinker with this some more, does anyone else also get this behavior?
Here's the place where I will be attempting my examples. https://github.com/codylane/testinfra-issue-gists/tree/master/issue-345
B U M P :shark: :shark: :shark:
Hi, I'm digging in this issue but I'm a bit confused because I'm not familiar with molecule and ansible. Can someone summarize (shortly) what the current state of this issue ?
IIUC, you run ansible through molecule which run a playbook with "vars_files", "roles" and role variables, and you expect to have a way in testinfra tests to get theses variables (= merge of host_vars, group_vars, vars_file, role default vars and role overriden vars) ?
testinfra doesn't have access to the playbook the machine was provisioned with, it only read inventory, ansible.cfg and host/groups vars.
I think it should be possible to add a playbook
parameter to host.ansible.get_variables()
which could read the playbook and merge the variables just like ansible do by calling the ansible playbook python API. Do you think it will solve your problem ?
Hi, I'm digging in this issue but I'm a bit confused because I'm not familiar with molecule and ansible. Can someone summarize (shortly) what the current state of this issue ?
Thanks for taking some time on this! We kind of got lost at sea :confounded:
IIUC, you run ansible through molecule which run a playbook with "vars_files", "roles" and role variables, and you expect to have a way in testinfra tests to get theses variables (= merge of host_vars, group_vars, vars_file, role default vars and role overriden vars) ?
Yes, you got it!
I think it should be possible to add a playbook parameter to host.ansible.get_variables() which could read the playbook and merge the variables just like ansible do by calling the ansible playbook python API. Do you think it will solve your problem ?
I think, basically, we decided that it would be the nicest if we could extend the get_variables()
interface, so this sounds like it would work! If I pass a bunch of vars into my playbook and I can pass my playbook into the function and get those vars, we're good.
First of all, I'd like to thank you too for spending your time on this.
I don't use molecule because I can't use docker on my work machine at the moment. During tests, I also like to use ansible variables though. But to test the roles in isolation, it's only possible for me to use the variables inside the role (defaults and vars). Therefore, I don't have a playbook for every role.
I took a glance at the ansible code and saw that the smallest unit for getting the variables is a play. A playbook could potentially hold multiple Plays which might make it difficult to load the correct variables from every playbook. We would probably need to specify at least the playbook and optionally which play we want to load.
For my specific use case, it would also be nice if I could just inline the necessary play data as a string or a dictionary and don't have to create a playbook for every role. Specifically, because I also run the tests of multiple roles combined to test a specific target host which had multiple plays run against it and the legacy roles I work with use the same variable name from time to time. Which would be problematic because some roles would overwrite values for other roles.
Yes, I think role vars are scoped to the role. So what do you think of this ? :
get_variables(playbook=<file or dict or yaml string>) --> would return merge of host_vars/group_vars and vars_file defined in playbook
get_variables(playbook=..., role=<string of a role name>) --> would return merge of host_vars/group_vars, vars_file AND role default vars + overriden role vars
It might be hard to write this, depending on the complexity of the ansible API, and by experience, it is so I think it's ok to not handle this for old ansible versions (and raise a NotImplementedError).
Right, apologies @barnabasJ, I see we have differing needs.
It might be hard to write this, depending on the complexity of the ansible API, and by experience, it is so I think it's ok to not handle this for old ansible versions (and raise a NotImplementedError).
<3
I've still not had time to do anything to make this happen but I promise I will QA test it :)