Find ansible_bender in group_vars

[timdiels]

When putting the ansible_bender var in group_vars/all.yml, ab fails to find it.

test.yml

- hosts: none
  tasks:
    - debug:
        msg: hi

group_vars/all.yml:

ansible_bender:
  base_image: python:3.5-stretch
  target_image:
    name: test

Running:

$ ansible-bender build test.yml
There was an error during execution: None is not of type 'string'

Failed validating 'type' in schema['properties']['base_image']:
    {'$id': '#/properties/base_image',
     'examples': ['fedora:29'],
     'pattern': '^(.*)$',
     'title': 'Base image',
     'type': 'string'}

On instance['base_image']:
    None

Environment is the same as my previous issue:

$ pip freeze
ansible-bender==0.5.2.dev1+g29c8d01
attrs==19.1.0
jsonschema==3.0.1
pyrsistent==0.14.11
PyYAML==5.1
six==1.12.0
tabulate==0.8.3
$ python --version
Python 3.7.2

Workaround: in test.yml vars: ansible_bender: '{{ _ansible_bender }}', in group_vars/all.yml define _ansible_bender as you normally would.

[TomasTomecek]

Honestly, this is what I don't like on Ansible :D you can perform one action in 12 different ways.

For loading variables, bender now supports only the vars section in a playbook and vars_files.

This should be implemented.

[timdiels]

Agreed, we all know what comes from "there's more than one way to do it" ;) The burden of figuring out which way is best and in which cases.

[timdiels]

Would it be possible to also allow other groups than all? E.g. currently I have:

  pre_tasks:
    # Put host in relevant groups (for the group vars)
    - group_by:
        key: managed
    - group_by:
        key: dev
        parents: managed

This allows me to access those group vars during the play, but I would like to be able to use e.g. dev group vars in my ansible_bender var too. I don't need to assign groups dynamically though, an ansible_bender_groups in which I list the groups it should be part of would do or maybe an inventory file. I would like to specify hierarchy in the groups too though, e.g. in this case managed: dev: hosts, so I can have some defaults in managed and override them in dev.

[timdiels]

A decent workaround for groups is to use vars_files:

  vars_files:
    - group_vars/all/vars.yml
    - group_vars/all/vault.yml
    - group_vars/managed.yml
    - group_vars/dev.yml

It still requires the _ansible_bender trick mentioned above though.

[TomasTomecek]

Okay, this is way beyond my Ansible-fu. Please, keep those comments coming for me to better understand your workflow.

Sadly, I won't have time to work on this in coming weeks (Red Hat Summit), but after that I'll try to look into it.

Thanks, @timdiels!

[quietjoy]

One option is to use the ansible-inventory command with the --graph option to pull out the hostvars and the groupvars for the particular playbook. It may not pull all the variables from the 20ish places you can put variables in ansible, but it gets close.

There are two complications.

• The output of the ansible-inventory --graph is in a weird format (not json, yaml or toml) so one would either need to parse the output or upstream the use of some other output format with the --graph option
• the ansible-inventory command, as the name suggest, requires an inventory file. To correctly pull out variables for a given play on a host, ansible needs to be aware of the host information. Perhaps adding an additional flag to ansible_bender so that you can specify the host file would be okay.

I think it would be a great addition to be able to say, define a common base container for a group of hosts in the group_vars and then specify specific tags for the individual containers in host_vars.

[TomasTomecek]

• The output of the ansible-inventory --graph is in a weird format (not json, yaml or toml) so one would either need to parse the output or upstream the use of some other output format with the --graph option
• the ansible-inventory command, as the name suggest, requires an inventory file. To correctly pull out variables for a given play on a host, ansible needs to be aware of the host information. Perhaps adding an additional flag to ansible_bender so that you can specify the host file would be okay.

I wonder if ansible-inventory has an API that we could utilize. That could make it simpler to use instead of invoking a CLI command & parsing the output.

I think it would be a great addition to be able to say, define a common base container for a group of hosts in the group_vars and then specify specific tags for the individual containers in host_vars.

Agreed!

[quietjoy]

I wonder if ansible-inventory has an API that we could utilize.

ansible-inventory is part of the main ansible codebase. I've been playing around with the ansible python API, and you can achieve the same thing. However, in the docs they are fairly explicit that this is not a stable API interface.

If you are willing to go that route and use the ansible python API directly, I think that using some of the highest level classes like InventoryManager, might be okay...? That may also introduce a bunch of complexity in keeping the project compatible with the different ansible releases.

[quietjoy]

I think it would be a great addition to be able to say, define a common base container for a group of hosts in the group_vars and then specify specific tags for the individual containers in host_vars.

Agreed!

My reading of the docs and examples is that all ansible-bender variables go under a top level ansible-bender variable. For example, in the sample playbook...

ansible_bender:
      base_image: docker.io/python:3-alpine
      target_image:
        name: a-very-nice-image

For this specific setup, I'm not sure ansible is capable of reading the variables if they are split up across host_vars and group_vars.

For example, if you have this setup in your ansible project...

group vars file

ansible_bender:
      base_image: docker.io/python:3-alpine

host vars file

ansible_bender:
      target_image:
        name: a-very-nice-image

The current ansible VariableManager.get_vars() method reads the variables by the order of precedence ansible defines. In other words, it will not combine the two ansible_bender values in hostvars and groupvars because it thinks there is another variable called ansible-bender that takes precedence. In the example above, only the value for ansible-bender in the hostvars would be returned from VariableManager.get_vars(). See this code.

I haven't found a way to just get group_vars without also reading the host_vars either directly with the ansible API or with the executables

Perhaps it would be possible to also take variables if they are prepended with a value such as ab_ or ansible_bender_

[TomasTomecek]

If you are willing to go that route and use the ansible python API directly, I think that using some of the highest level classes like InventoryManager, might be okay...? That may also introduce a bunch of complexity in keeping the project compatible with the different ansible releases.

To be honest, in the callback that is implemented in this codebase, we already access some Ansible internals.

But you're right that if the API is changing a lot between releases, we shouldn't do that and instead utilize the ansible-inventory command.

The current ansible VariableManager.get_vars() method reads the variables by the order of precedence ansible defines. In other words, it will not combine the two ansible_bender values in hostvars and groupvars because it thinks there is another variable called ansible-bender that takes precedence. In the example above, only the value for ansible-bender in the hostvars would be returned from VariableManager.get_vars(). See this code.

I haven't found a way to just get group_vars without also reading the host_vars either directly with the ansible API or with the executables

It would be for the best if we stayed consistent with Ansible as much as possible.

What would you suggest as the best solution?

Thank you for your thorough research!

Perhaps it would be possible to also take variables if they are prepended with a value such as ab_ or ansible_bender_

Sounds solid to me.

[quietjoy]

What would you suggest as the best solution?

I think that we should maintain the current functionality (using the typical ansible precedence for variables) if one wants to put all ansible_bender variable in one place.

Perhaps we could also allow these variables to be prepended with ansible_bender*

I see three main groups of ansible_bender variables:

• base_image
• working_container
• target_image

so that you could have the following breakdown of variables for host host1 in group group1

host_vars/host1

---
ansible_bender_target_image:
   name: a-very-nice-image
        working_dir: /src
        environment:
          FILE_TO_PROCESS: README.md

group_vars/group1

---
ansible_bender_base_image: docker.io/python:3-alpine

playbook.yml

- name: host1 configuration
  vars:
    ansible_bender_working_container:
        volumes:
          - '{{ playbook_dir }}:/src:Z'

So, in summary, if the variable ansible_bender is found - use it. If not, look for the pre-pended ansible_bender* variables. I am certainly open to suggestions on how to approach this problem.

[TomasTomecek]

I like the proposal, clear and logical. We also need to document the hierarchy so it's easy to understand by everyone.

During implementation, we should come up with several test cases to make sure the inheritance works correctly.