amazon.aws
amazon.aws copied to clipboard
ansible-collections
ec2_instance - duplicate security_groups returns bad error message when no vpc specified.
SUMMARY
Under certain circumstances ec2.py reports "The following group names are not valid:..." when the real problem the is named group does not exist, or duplicate group names were found. This sends the user into the mire that is force_valid_group_names, never to return, at least not sane anyway...
ISSUE TYPE
- Bug Report
COMPONENT NAME
ec2_instance
ANSIBLE VERSION
ansible 2.10.7
CONFIGURATION
ANSIBLE_PIPELINING(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxx.cfg) = True
ANSIBLE_SSH_ARGS(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxx.cfg) = -o ServerAliveInterval=30 -o ControlMaster=auto -o ControlPersist=30m -o StrictHostKeyChecking=no
ANSIBLE_SSH_CONTROL_PATH(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxxx.cfg) = %(directory)s/ssh-%%C
DEFAULT_HOST_LIST(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxxx.cfg) = ['/home/robert/gitrep/cp_ansible/inventory']
DEFAULT_MANAGED_STR(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxx.cfg) = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
DEFAULT_PRIVATE_KEY_FILE(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxxx.cfg) = /home/robert/.ssh/xxxxxxxxxxx
DEFAULT_REMOTE_USER(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxxxx.cfg) = ubuntu
DEFAULT_ROLES_PATH(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxxxx.cfg) = ['/home/robert/gitrep/cp_ansible/roles/internal', '/home/robert/gitrep/cp_ansible/roles/external', '/etc/a
DEFAULT_VAULT_PASSWORD_FILE(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxxxx.cfg) = xxxxxxx
INVENTORY_ENABLED(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxxx.cfg) = ['auto']
TRANSFORM_INVALID_GROUP_CHARS(/home/robert/gitrep/cp_ansible/plays/ansible-xxxxxx.cfg) = never
OS / ENVIRONMENT
ubuntu linux 20.04
STEPS TO REPRODUCE
On AWS you will need two security groups in different VPC's, with the same names
Using ec2_instance update the security groups with ansible, but do not include a vpc_id or a vpc_subnet_id
(Apologies this is in a loop)
- name: Update instances.
ec2_instance:
region: "{{ vpc.region }}"
filters:
"instance_id": "{{item}}"
"tag:Name": "{{ outer_item.0.instance_tags.Name }}"
"tag:Zone": "{{ outer_item.0.instance_tags.Zone }}"
instance-state-name:
- running
security_groups: "{{ outer_item.0.groups }}"
# vpc_subnet_id: "{{ outer_item.1.subnets|map(attribute='id')|list|random }}"
register: ec2_results_inner
with_items: "{{existing_instances.instances|list |map(attribute='instance_id') | list}}"
this fails in line 588, in get_ec2_security_group_ids_from_names\nValueError: The following group names are not valid: ....outer_item.0.groups names....\n"
Since the names have hyphens this sent me on the path of force_valid_group_names, and why it appeared not to work.
The real error is the code fails to match all the group names (as there are duplicates) and then puts out an unhelpful message.
- name: Update instances.
ec2_instance:
region: "{{ vpc.region }}"
filters:
"instance_id": "{{item}}"
"tag:Name": "{{ outer_item.0.instance_tags.Name }}"
"tag:Zone": "{{ outer_item.0.instance_tags.Zone }}"
instance-state-name:
- running
security_groups: "{{ outer_item.0.groups }}"
vpc_subnet_id: "{{ outer_item.1.subnets|map(attribute='id')|list|random }}"
register: ec2_results_inner
with_items: "{{existing_instances.instances|list |map(attribute='instance_id') | list}}"
This works as expected. All groups are matched so in line 587 len(still_unmatched) ==0
EXPECTED RESULTS
When matching group names all duplicates should be matched too. Even though the duplicates may be in other VPC's the name is still valid input for the instance since at least one of the duplicate groups are in the same vpc. Alternatively output a more useful error message
ACTUAL RESULTS
As above the error message is output and (very) unhelpful.
Files identified in the description:
- [
scripts/inventory/ec2.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/scripts/inventory/ec2.py)
If these files are inaccurate, please update the component name section of the description or use the !component bot command.
It's odd that it's telling you that a variable name is not valid, generally the module shouldn't see the variable names. Please could you provide the output while using -vvvv it would be helpful to see what the module thinks its actually being passed.
Files identified in the description:
If these files are inaccurate, please update the component name section of the description or use the !component bot command.
It doesnt actually output ....outer_item.0.groups names...., it outputs the contents. I just used the ...var.... to represent the contents to avoid exposing some names. Was actually The following group names are not valid: awx_dev_xxx-inbound_internal, bastionhost_dev_xxx-inbound_internal, nginx_dev_xxx-inbound These three exist in multiple VPC's in this region
TASK [aws.ec2 : Update instances.] *****************************************************************************************************************************************************
task path: /home/xxxxxx/gitrep/cp_ansible/plays/roles/aws.ec2/tasks/update.yml:3
redirecting (type: modules) ansible.builtin.ec2_instance to community.aws.ec2_instance
Using module file /usr/local/lib/python3.6/dist-packages/ansible_collections/community/aws/plugins/modules/ec2_instance.py
Pipelining is enabled.
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: xxxxxx
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 && sleep 0'
The full traceback is:
Traceback (most recent call last):
File "<stdin>", line 102, in <module>
File "<stdin>", line 94, in _ansiballz_main
File "<stdin>", line 40, in invoke_module
File "/usr/lib/python3.6/runpy.py", line 205, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/usr/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py", line 1795, in <module>
File "/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py", line 1780, in main
File "/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py", line 1607, in ensure_present
File "/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py", line 1589, in handle_existing
File "/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py", line 1335, in diff_instance_and_params
File "/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py", line 1138, in discover_security_groups
File "/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/ec2.py", line 588, in get_ec2_security_group_ids_from_names
ValueError: The following group names are not valid: awx_dev_xxx-inbound_internal, bastionhost_dev_xxx-inbound_internal, nginx_dev_xxx-inbound
failed: [localhost] (item=i-05174a732e7ad9848) => {
"ansible_loop_var": "item",
"changed": false,
"item": "i-05174a732e7ad9848",
"module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py\", line 1795, in <module>\n File \"/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py\", line 1780, in main\n File \"/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py\", line 1607, in ensure_present\n File \"/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py\", line 1589, in handle_existing\n File \"/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py\", line 1335, in diff_instance_and_params\n File \"/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_instance.py\", line 1138, in discover_security_groups\n File \"/tmp/ansible_ec2_instance_payload_bk0bnr6m/ansible_ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/ec2.py\", line 588, in get_ec2_security_group_ids_from_names\nValueError: The following group names are not valid: awx_dev_xxx-inbound_internal, bastionhost_dev_xxx-inbound_internal, nginx_dev_xxx-inbound\n",
"module_stdout": "",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
"rc": 1
}
Hi @rob42. I'm not able to reproduce this with the latest collection versions, I've got 2 VPCs in us-west-2 and I've created a security group in each named duplicate_test and I'm modifying an existing instance. Can you check what versions of community.aws (for the ec2_instance module) and amazon.aws (for the ec2 module_util) you're running with ansible-galaxy collection list?
1.5 was just released for both collections last week and there were some other patches for ec2_instance in that, if you can try testing with those versions that would be great too.
Looks like I have v1.4 amazon collections
Collection Version
amazon.aws 1.4.1
ansible.netcommon 1.5.0
ansible.posix 1.2.0
ansible.utils 2.0.1
ansible.windows 1.4.0
arista.eos 1.3.0
awx.awx 17.1.0
azure.azcollection 1.4.0
check_point.mgmt 2.0.0
chocolatey.chocolatey 1.0.2
cisco.aci 2.0.0
cisco.asa 1.0.4
cisco.intersight 1.0.11
cisco.ios 1.3.0
cisco.iosxr 1.2.1
cisco.meraki 2.2.1
cisco.mso 1.1.0
cisco.nso 1.0.3
cisco.nxos 1.4.0
cisco.ucs 1.6.0
cloudscale_ch.cloud 2.1.0
community.aws 1.4.0
community.azure 1.0.0
community.crypto 1.5.0
community.digitalocean 1.0.0
community.docker 1.3.0
community.fortios 1.0.0
community.general 2.2.0
community.google 1.0.0
community.grafana 1.2.0
community.hashi_vault 1.1.2
community.hrobot 1.1.1
community.kubernetes 1.2.0
community.kubevirt 1.0.0
community.libvirt 1.0.1
community.mongodb 1.2.1
community.mysql 1.3.0
community.network 2.0.1
community.okd 1.1.0
community.postgresql 1.1.1
community.proxysql 1.0.0
community.rabbitmq 1.0.3
community.routeros 1.1.0
community.skydive 1.0.0
community.sops 1.0.5
community.vmware 1.8.0
community.windows 1.3.0
community.zabbix 1.2.0
containers.podman 1.4.4
cyberark.conjur 1.1.0
cyberark.pas 1.0.5
dellemc.openmanage 3.1.0
dellemc.os10 1.1.1
dellemc.os6 1.0.7
dellemc.os9 1.0.4
f5networks.f5_modules 1.8.1
fortinet.fortimanager 2.0.1
fortinet.fortios 1.1.9
frr.frr 1.0.3
gluster.gluster 1.0.1
google.cloud 1.0.2
hetzner.hcloud 1.2.1
ibm.qradar 1.0.3
infinidat.infinibox 1.2.4
inspur.sm 1.1.2
junipernetworks.junos 1.3.0
kubernetes.core 1.2.0
mellanox.onyx 1.0.0
netapp.aws 20.9.0
netapp.elementsw 20.11.0
netapp.ontap 21.3.1
netapp_eseries.santricity 1.1.0
netbox.netbox 2.1.0
ngine_io.cloudstack 2.0.0
ngine_io.exoscale 1.0.0
ngine_io.vultr 1.1.0
openstack.cloud 1.3.0
openvswitch.openvswitch 1.2.0
ovirt.ovirt 1.3.1
purestorage.flasharray 1.6.2
purestorage.flashblade 1.4.0
sensu.sensu_go 1.9.1
servicenow.servicenow 1.0.4
splunk.es 1.0.2
t_systems_mms.icinga_director 1.15.0
theforeman.foreman 1.5.1
vyos.vyos 1.1.1
wti.remote 1.0.1
@rob42 big thank you for this one, i was stuck for hours on this one! 😃
My problem was NOT because it was duplicate security group names but just because the instance's VPC was not a default VPC.
My problem was NOT because it was duplicate security group names but just because the instance's VPC was not a default VPC.
Just ran into this, the when creating and ec2_group I neglected to specify the VPC, so it went into the default one, hence making the reference to it in a different VPC invalid. Would be nice to have a 'does not exist in this VPC' error message but ho-hum.
