Cannot create Elastic Search cluster using advanced security options

[rogozind]

Summary

This is ansible fragment from the code to create cluster:

        advanced_security_options:
          enabled: true
          internal_user_database_enabled: true
          master_user_options:
            master_user_name: "{{ opensearch_user }}"
            master_user_password: "{{ opensearch_password }}"

This is the error I get: File "/usr/local/Cellar/[email protected]/3.8.7/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/folders/4p/p4gsm16109d0p78txhvjc2mw0000gn/T/ansible_community.aws.opensearch_payload_23ucyoth/ansible_community.aws.opensearch_payload.zip/ansible_collections/community/aws/plugins/modules/opensearch.py", line 1500, in File "/var/folders/4p/p4gsm16109d0p78txhvjc2mw0000gn/T/ansible_community.aws.opensearch_payload_23ucyoth/ansible_community.aws.opensearch_payload.zip/ansible_collections/community/aws/plugins/modules/opensearch.py", line 1494, in main File "/var/folders/4p/p4gsm16109d0p78txhvjc2mw0000gn/T/ansible_community.aws.opensearch_payload_23ucyoth/ansible_community.aws.opensearch_payload.zip/ansible_collections/community/aws/plugins/modules/opensearch.py", line 1232, in ensure_domain_present File "/var/folders/4p/p4gsm16109d0p78txhvjc2mw0000gn/T/ansible_community.aws.opensearch_payload_23ucyoth/ansible_community.aws.opensearch_payload.zip/ansible_collections/community/aws/plugins/modules/opensearch.py", line 956, in set_advanced_security_options KeyError: 'MasterUserOptions'

I think the code is trying to access MasterUserOptions key w/o setting it empty dictionary first.

Issue Type

Bug Report

Component Name

community.aws.opensearch

Ansible Version

$ ansible --version
ansible [core 2.12.4]
  config file = /Users/dima/GIT/devops/ansible/ansible.cfg
  configured module search path = ['/Users/dima/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  ansible collection location = /Users/dima/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.8.7 (default, Dec 30 2020, 10:14:55) [Clang 12.0.0 (clang-1200.0.32.28)]
  jinja version = 2.11.3
  libyaml = True

Collection Versions

$ ansible-galaxy collection list

/Users/dima/.ansible/collections/ansible_collections

Collection Version

---

amazon.aws 5.0.2
community.aws 5.0.0

/usr/local/lib/python3.8/site-packages/ansible_collections

Collection Version

---

amazon.aws 2.2.0
ansible.netcommon 2.6.1
ansible.posix 1.3.0
ansible.utils 2.5.2
ansible.windows 1.9.0
arista.eos 3.1.0
awx.awx 19.4.0 azure.azcollection 1.12.0 check_point.mgmt 2.3.0
chocolatey.chocolatey 1.2.0
cisco.aci 2.2.0
cisco.asa 2.1.0
cisco.intersight 1.0.18 cisco.ios 2.8.1
cisco.iosxr 2.9.0
cisco.ise 1.2.1
cisco.meraki 2.6.1
cisco.mso 1.4.0
cisco.nso 1.0.3
cisco.nxos 2.9.1
cisco.ucs 1.8.0
cloud.common 2.1.0
cloudscale_ch.cloud 2.2.1
community.aws 2.4.0
community.azure 1.1.0
community.ciscosmb 1.0.4
community.crypto 2.2.4
community.digitalocean 1.16.0 community.dns 2.0.9
community.docker 2.3.0
community.fortios 1.0.0
community.general 4.7.0
community.google 1.0.0
community.grafana 1.3.3
community.hashi_vault 2.4.0
community.hrobot 1.2.3
community.kubernetes 2.0.1
community.kubevirt 1.0.0
community.libvirt 1.0.2
community.mongodb 1.3.3
community.mysql 2.3.5
community.network 3.1.0
community.okd 2.1.0
community.postgresql 1.7.1
community.proxysql 1.3.1
community.rabbitmq 1.1.0
community.routeros 2.0.0
community.sap 1.0.0
community.skydive 1.0.0
community.sops 1.2.1
community.vmware 1.18.0 community.windows 1.9.0
community.zabbix 1.5.1
containers.podman 1.9.3
cyberark.conjur 1.1.0
cyberark.pas 1.0.13 dellemc.enterprise_sonic 1.1.0
dellemc.openmanage 4.4.0
dellemc.os10 1.1.1
dellemc.os6 1.0.7
dellemc.os9 1.0.4
f5networks.f5_modules 1.15.0 fortinet.fortimanager 2.1.4
fortinet.fortios 2.1.4
frr.frr 1.0.3
gluster.gluster 1.0.2
google.cloud 1.0.2
hetzner.hcloud 1.6.0
hpe.nimble 1.1.4
ibm.qradar 1.0.3
infinidat.infinibox 1.3.3
infoblox.nios_modules 1.2.1
inspur.sm 1.3.0
junipernetworks.junos 2.10.0 kubernetes.core 2.3.0
mellanox.onyx 1.0.0
netapp.aws 21.7.0 netapp.azure 21.10.0 netapp.cloudmanager 21.15.0 netapp.elementsw 21.7.0 netapp.ontap 21.17.3 netapp.storagegrid 21.10.0 netapp.um_info 21.8.0 netapp_eseries.santricity 1.3.0
netbox.netbox 3.6.0
ngine_io.cloudstack 2.2.3
ngine_io.exoscale 1.0.0
ngine_io.vultr 1.1.1
openstack.cloud 1.7.2
openvswitch.openvswitch 2.1.0
ovirt.ovirt 1.6.6
purestorage.flasharray 1.12.1 purestorage.flashblade 1.9.0
sensu.sensu_go 1.13.0 servicenow.servicenow 1.0.6
splunk.es 1.0.2
t_systems_mms.icinga_director 1.28.0 theforeman.foreman 2.2.0
vyos.vyos 2.8.0
wti.remote 1.0.3
office:ansible dima$

AWS SDK versions

$ pip show boto boto3 botocore

office:ansible dima$ pip3 show boto boto3 botocore Name: boto Version: 2.49.0 Summary: Amazon Web Services Library Home-page: https://github.com/boto/boto/ Author: Mitch Garnaat Author-email: [email protected] License: MIT Location: /usr/local/lib/python3.8/site-packages Requires: Required-by:

Name: boto3 Version: 1.24.89 Summary: The AWS SDK for Python Home-page: https://github.com/boto/boto3 Author: Amazon Web Services Author-email: None License: Apache License 2.0 Location: /usr/local/lib/python3.8/site-packages Requires: botocore, jmespath, s3transfer Required-by:

Name: botocore Version: 1.27.89 Summary: Low-level, data-driven core of boto 3. Home-page: https://github.com/boto/botocore Author: Amazon Web Services Author-email: None License: Apache License 2.0 Location: /usr/local/lib/python3.8/site-packages Requires: python-dateutil, urllib3, jmespath Required-by: s3transfer, boto3 office:ansible dima$

Configuration

$ ansible-config dump --only-changed

OS / Environment

No response

Steps to Reproduce

Just create the cluster with teh advanced security:

    advanced_security_options:
      enabled: true
      internal_user_database_enabled: true
      master_user_options:
        master_user_name: "{{ opensearch_user }}"
        master_user_password: "{{ opensearch_password }}"

Expected Results

Ansible should not crash

Actual Results

Code of Conduct

• [X] I agree to follow the Ansible Code of Conduct

[gionn]

Hello, same issue here:

    - name: Create OpenSearch domain for dev environment, no zone awareness, no dedicated masters
      community.aws.opensearch:
        domain_name: "{{ domain_name }}"
        engine_version: Elasticsearch_7.10
        cluster_config:
          instance_type: "t2.small.search"
          instance_count: 2
          zone_awareness: false
          dedicated_master: false
        ebs_options:
          ebs_enabled: true
          volume_type: "gp2"
          volume_size: 10
        advanced_security_options:
          enabled: true
          internal_user_database_enabled: false
          master_user_options:
            master_user_name: myusername
            master_user_password: asecurepassword

Fails with:

KeyError: 'MasterUserOptions'
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 107, in <module>\n  File \"<stdin>\", line 99, in _ansiballz_main\n  File \"<stdin>\", line 47, in invoke_module\n  File \"/Users/Giovanni.Toraldo/.pyenv/versions/3.9.13/lib/python3.9/runpy.py\", line 225, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/Users/Giovanni.Toraldo/.pyenv/versions/3.9.13/lib/python3.9/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/Users/Giovanni.Toraldo/.pyenv/versions/3.9.13/lib/python3.9/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/var/folders/l4/rr1s57q973ggg6ylq20bqy6h0000gq/T/ansible_community.aws.opensearch_payload_nvm9_wbt/ansible_community.aws.opensearch_payload.zip/ansible_collections/community/aws/plugins/modules/opensearch.py\", line 1500, in <module>\n  File \"/var/folders/l4/rr1s57q973ggg6ylq20bqy6h0000gq/T/ansible_community.aws.opensearch_payload_nvm9_wbt/ansible_community.aws.opensearch_payload.zip/ansible_collections/community/aws/plugins/modules/opensearch.py\", line 1494, in main\n  File \"/var/folders/l4/rr1s57q973ggg6ylq20bqy6h0000gq/T/ansible_community.aws.opensearch_payload_nvm9_wbt/ansible_community.aws.opensearch_payload.zip/ansible_collections/community/aws/plugins/modules/opensearch.py\", line 1232, in ensure_domain_present\n  File \"/var/folders/l4/rr1s57q973ggg6ylq20bqy6h0000gq/T/ansible_community.aws.opensearch_payload_nvm9_wbt/ansible_community.aws.opensearch_payload.zip/ansible_collections/community/aws/plugins/modules/opensearch.py\", line 952, in set_advanced_security_options\nKeyError: 'MasterUserOptions'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

[alinabuzachis]

@gionn @rogozind Thank you for reporting this. Would anyone be willing to to open a PR to fix this bug?

[rogozind]

I am not sure if this is the right fix but this patch solved it for me:

./community/aws/plugins/modules/opensearch.py:

          master_user_opts = advanced_security_opts.get("master_user_options")
          if master_user_opts is not None:
+             advanced_security_config["MasterUserOptions"] = {}
              if master_user_opts.get("master_user_arn") is not None:
                  advanced_security_config["MasterUserOptions"][

[gionn]

@rogozind looks good! I've opened a PR with that change if you don't mind

[gionn]

🚀

Thanks all!