Possibility to use a loop over controller items?

[nodje]

It's nice to benefit from the infra.controller abstraction over the ansible.controller collection, but I'm reaching a density in my filetree_read source where a loop notion becomes necessary.

I have enough object to create that I'd need a mecanism like:

controller_organizations:
  - name: "{{ item }}"
    instance_groups: "{{ common_instance_groups }}"
    default_environment: "{{ common_default_execution_environment }}"
    galaxy_credentials: "{{ common_galaxy_credentials }}"
    loop:
     - org_list

rather than to have to duplicate the block 50 times over.

But I don't see how this would be possible in the filetree_read source variables.

Right now it seems that my only option to be able to loop over list of items would be to go back a layer down and start to use ansible.controller.

Some controllers like controller_roles have this use case kind of embedded as it can be applied to both team and teams list, organization and organizations list, etc.

Maybe adding this possibility to all controller object could be a solution, so that a list of names could be used as an entry for any type of object?

[ivarmu]

I'm not sure to get you... The code you are showing seems to loop over many organizations... but the filetree_read expects to treat only one organization at a time. The same playbook can be executed multiple times only changing the input variables to manage the desired organization: dir_orgs_vars and orgs.

I've removed the bug label as I don't see something that has stopped working... for me is more an enhancement.

[nodje]

Correct, it's an enhancement requirement. As there were very little activity in enhancement I thought it was better to stick to issue. Sorry about that.

I used Organization as an example here. But the requirement is for all kinds of controller.

My need is, rather than repeating the block

controller_organizations:
  - name: "Org1"
    instance_groups: "{{ common_instance_groups }}"
    default_environment: "{{ common_default_execution_environment }}"
    galaxy_credentials: "{{ common_galaxy_credentials }}"

50 times to apply an identical configuration, I'd much rather have a source list of Org names

Org_name_list:
  - org1
  ...
  - org50

and have a way to do something like this:

controller_organizations:
  - name: "{{ item }}"
    instance_groups: "{{ common_instance_groups }}"
    default_environment: "{{ common_default_execution_environment }}"
    galaxy_credentials: "{{ common_galaxy_credentials }}"
    loop:
     - Org_name_list

This of course isn't possible with the actual system. But it'd help avoid code duplication, and is in the end necessary for code maintenance as the code base grow in size.

This has been made possible in the controller_roles by adding teams on top of team, users on top of user etc.

The simplest way to enable this would probably be to change name::str into names::str as has been done in the controller_role.

So that this snippet would work on a set of Org:

controller_organizations:
  - names: "{{ Org_name_list }}"
    instance_groups: "{{ common_instance_groups }}"
    default_environment: "{{ common_default_execution_environment }}"
    galaxy_credentials: "{{ common_galaxy_credentials }}"

then of course the same would be needed for controller_teams, controller_inventories, controller_credentials ... most of the AWX resources.

[ivarmu]

I have some things to comment here...

• The modifications proposed are not simple and can be a hard work for each object type. We must be sure that this is really required and have sense to be implemented.
• If you name equally a Job Template on different organizations, your CasC code will presumably fail, as it expects the object names to be unique across the AAP instance, so the object naming should contain, for example, the name of the organization it belong to, as part of the object's name, to avoid duplicated names across different organizations.
• You can develop your playbooks in different ways... personally, I've ever created them to be valid for only one organization, and I've reused them running them multiple times with different input parameters, being the organization one of them. You can run it multiple times from the command line, one time for each organization. Would it feet your needs? An example of this command follows, where you can substitute the ORGNAME environment variable to your needs:

  ansible-navigator run casc_ctrl_config.yml -i inventory -l dev -e "{orgs: '${ORGNAME}', dir_orgs_vars: orgs_vars, env: dev}" -m stdout --eei quay.io/automationiberia/aap/ee-casc --vault-password-file .vault_password
  

[nodje]

thanks for your comment. Having a different playbook per org seems to be a good strategy, though I don't think I could use it for our situation.

I'll share my specific use case to give you some perspective.

We configure a tower instance from scratch with our infra.controller_configuration based playbook.

We do it in an idempotent spirit, meaning we run the same playbook with a CI ever as many time as needed to keep the targets env in sync. (even though we're struggling with deleting object, we mostly do it manually or with state:absent when possible as object_diff proves difficult to use for now)

An important sub Use case that triggered this request is that we on-board regularly new department of our business, under the form of a dedicated Org.

When we onboard a new dept XXXX, we have to perform the following actions:

• add an Org XXXX
• add a Team XXXX belonging to Org XXXX
• add a set of roles to the Team XXXX that applies to Org XXXX (the set of roles is a list var)
• add a team_org_map item (team: XXXX organization: XXXX) to SAML setting SOCIAL_AUTH_SAML_TEAM_ATTR

For now we have to duplicate this basic config for each new org. As we reach ~40 it's becoming hard to maintain. It doesn't seem necessary to duplicate this configs as they are all the same. It should be a "function" that gets the Org/Team name in argument.

That's the main issue for us, but we have a lot of configs that are identical and that we have to duplicate.

Now thinking about it I can see how I could have a subset with only the Orga/Team/Role/Setting and loop over a call to filetree_read+dispatch, with the Org/Team name as a var, from the main playbook. Instead of a single unique call for all controller source as we naturally do.

That would work but it doesn't feel proper IMO at the moment. Actually the SAML setting is an issue as it's a single var, you can't add to it, you have to set the entire value. (this would be a good new feature as well)

Also it's hard to keep track between controller dependencies as the controller source base grows.

All in all, it feels like a good python SDK for Awx/Tower would help a lot :) There use to be one as I remember. I'm not sure why Red Hat focuses on playbook only as the main way to configure/use AAP honestly.

[Tompage1994]

I'm not sure it will fix your problem, but it is possible to make use of YAML Anchors as part of your definition. You can then set out your example above like so:

common_instance_groups: my_ig
common_default_execution_environment: my_ee
common_galaxy_credentials: my_cred

__org_default_opts: &org_defaults
  instance_groups: "{{ common_instance_groups }}"
  default_environment: "{{ common_default_execution_environment }}"
  galaxy_credentials: "{{ common_galaxy_credentials }}"

controller_organizations:
  - <<: *org_defaults
    name: org1
  - <<: *org_defaults
    name: org2
  - <<: *org_defaults
    name: org3
    default_environment: override_ee  # This is overrideing the standard value you set

The resultant variable output then is:

controller_organizations:
- default_environment: my_ee
  galaxy_credentials: my_cred
  instance_groups: my_ig
  name: org1
- default_environment: my_ee
  galaxy_credentials: my_cred
  instance_groups: my_ig
  name: org2
- default_environment: override_ee
  galaxy_credentials: my_cred
  instance_groups: my_ig
  name: org3

This obviously doesn't get it down to a nice singular list but it reduces the vast majority of the repeated code. You'd probably also want to make sure there is sufficient documentation around what's going on here as someone who hasn't come across this notation before may not understand what's going on

[nodje]

thanks a bunch @Tompage1994, it's a freaking good idea, it totally escaped me to use anchors. I think I somehow considered the controller declarations a 'proprietary' syntax. But all right, no, it's simply YAML and can be interpreted as such with all the power of its syntax.

This will help me come a long way from where we are now, for the configuration code duplication part.

But as you mentioned, I still won't be able to use a list type variable and apply a controller conf to each of its member though. Maybe I'd be worth for your team to study for which controller it'd be easy to migrate to a 'plural name' support, as is done for the role controller. I see it as a nice additional feature, but maybe infra.controller_configuration is not meant to be used this way.