aws_ec2 inventory does not support configuration through environment variables

[dnmvisser]

Summary

I've successfully migrated from ec2.py to the new EC2 inventory source. This works great and we can get the same functionality (or better) with just a tiny configuration file.

But, it is not possible anymore to configure it using environment variables.

We use shell wrappers to deploy things, which allowed us to use a single ec2.ini file, and narrow things down by constructs like:

EC2_INSTANCE_FILTERS="tag:Environment=${env}" ansible-playbook -i ec2.py ....

With aws_ec2 that flexibility is now gone, as everything (apart from the credentials) has to be hard coded in the aws_ec2.yml file. For example we now have to have different config files for each environment:

---
plugin: amazon.aws.aws_ec2
regions: eu-central-1
filters:
    tag:Environment:
    - production
hostnames:
- tag:InventoryHostname

---
plugin: amazon.aws.aws_ec2
regions: eu-central-1
filters:
    tag:Environment:
    - acceptance
hostnames:
- tag:InventoryHostname

etc etc.

I was hoping this would work, but it doesn't:

---
plugin: amazon.aws.aws_ec2
regions: eu-central-1
filters:
    tag:Environment:
    - "{{ env }}"
hostnames:
- tag:InventoryHostname

Issue Type

Bug Report

Component Name

plugins/inventory/aws_ec2.py

Ansible Version

ansible [core 2.12.2]
  config file = /Users/dick.visser/gitlab/project8/data/ansible.cfg
  configured module search path = ['/Users/dick.visser/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/dick.visser/gitlab/project8/venv/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/dick.visser/gitlab/project8/cache
  executable location = /Users/dick.visser/gitlab/project8/venv/bin/ansible
  python version = 3.9.9 (main, Jan 25 2022, 23:02:06) [Clang 13.0.0 (clang-1300.0.29.30)]
  jinja version = 3.0.3
  libyaml = True

Collection Versions

# /Users/dick.visser/git/project8/ansible_collections
Collection            Version
--------------------- -------
amazon.aws            2.1.0  
ansible.netcommon     2.5.0  
ansible.posix         1.3.0  
ansible.utils         2.5.0  
community.aws         2.2.0  
community.crypto      2.2.0  
community.dns         2.0.6  
community.docker      2.1.1  
community.general     4.4.0  
community.grafana     1.3.0  
community.hashi_vault 2.2.0  
community.kubernetes  2.0.1  
community.kubevirt    1.0.0  
community.network     3.0.0  
community.postgresql  1.6.1  
community.zabbix      1.5.1  
containers.podman     1.9.1  
kubernetes.core       2.2.3  

AWS SDK versions

Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: [email protected]
License: MIT
Location: /Users/dick.visser/gitlab/project8/venv/lib/python3.9/site-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.21.3
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /Users/dick.visser/gitlab/project8/venv/lib/python3.9/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.24.3
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /Users/dick.visser/gitlab/project8/venv/lib/python3.9/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: awscli, boto3, s3transfer

Configuration

OLLECTIONS_PATHS(/Users/dick.visser/gitlab/project8/data/ansible.cfg) = ['/Users/dick.visser/gitlab/project8/cache']
DEFAULT_FILTER_PLUGIN_PATH(/Users/dick.visser/gitlab/project8/data/ansible.cfg) = ['/Users/dick.visser/gitlab/project8/playbooks/filter_plugins']
DEFAULT_FORKS(/Users/dick.visser/gitlab/project8/data/ansible.cfg) = 10
DEFAULT_GATHER_TIMEOUT(/Users/dick.visser/gitlab/project8/data/ansible.cfg) = 60
DEFAULT_LOAD_CALLBACK_PLUGINS(/Users/dick.visser/gitlab/project8/data/ansible.cfg) = True
DEFAULT_NO_TARGET_SYSLOG(/Users/dick.visser/gitlab/project8/data/ansible.cfg) = True
DEFAULT_STDOUT_CALLBACK(/Users/dick.visser/gitlab/project8/data/ansible.cfg) = yaml
DEFAULT_TIMEOUT(/Users/dick.visser/gitlab/project8/data/ansible.cfg) = 30
INTERPRETER_PYTHON(/Users/dick.visser/gitlab/project8/data/ansible.cfg) = /usr/bin/env python3
RETRY_FILES_ENABLED(/Users/dick.visser/gitlab/project8/data/ansible.cfg) = False

OS / Environment

macOS 12.2.1

Steps to Reproduce

Try to use AWS EC2 inventory configuration like this:

---
plugin: amazon.aws.aws_ec2
regions: eu-central-1
filters:
    tag:Environment:
    - "{{ env }}"
hostnames:
- tag:InventoryHostname

and/or run it like this:

EC2_INSTANCE_FILTERS="tag:Environment=${env}" ansible-playbook -i ec2.py ....

Expected Results

I expected the inventory to further filter based on the env environment variable

Actual Results

No additional filtering applied.

Code of Conduct

• [X] I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• [plugins/inventory/aws_ec2.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/inventory/aws_ec2.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[alinabuzachis]

@dnmvisser Thank you for raising this. We will further investigate and get back to you.

[markuman]

The current aws_ec2.py dynamic inventory does not handle custom environment variables like the old one did:

https://github.com/ansible-community/contrib-scripts/blob/main/inventory/ec2.py#L496

So at the moment, you can only rely on env variables that are interpretated by boto3. And imo, boto3 does not offer such things.

I was hoping this would work, but it doesn't:

---
plugin: amazon.aws.aws_ec2
regions: eu-central-1
filters:
    tag:Environment:
    - "{{ env }}"
hostnames:
- tag:InventoryHostname

afaik that doesn't work, because you must use -e env=production to use {{ env }} in plays. So you can use it in playbooks, not in inventories.

but the Instance tags are available as host_vars. That means if you do

ansible-playbook -i some_inv.aws_ec2.yml my_play.yml  -e "Environment=${env}" 

and adjust your play a little

---
- hosts: all

  tasks:
    - name: my tasks
      when: tags.Environment == Environment
      block:
       ...

you can reach the same result - imo.

[goneri]

I don't feel like it's a good idea to implement this. This will create yet another configuration source which will ultimately increase the complexity. Also the original key name is rather counter intuitive now we've got the include_filters and exclude_filters mechanism in place.

[goneri]

@dnmvisser you can also generate the inventory configuration from a template and set the key using include_filters: { lookup('env', 'EC2_INSTANCE_FILTERS') }}