Correct use of security_group with azure_rm_networkinterface unclear from documentation

[LHurst-JM]

SUMMARY

I think there are a couple of issues with the documentation for using security_group with a NIC.

1. The first is that if create_with_security_group is set to False, the security group is cleared - that is existing security groups are removed from the interface. This is not clear from the documentation, which says "Whether a security group should be be created with the NIC. If this flag set to True and no security_group set, a default security group will be created.". I think a sentence should be added that reads "If this flag is set to False any existing security group will be removed from the NIC."

2. If create_with_security_group is set to False, any existing security group set using security_group will not be set (the NIC's existing security_group will in fact be cleared). Again, this is unclear from the documentation which just describes create_with_security_group as controlling the creation of a new security group ("Whether a security group should be be created with the NIC." (amphasis mine)). The above wording change (1.) would also clarify this behaviour (i.e. security_group only does anything with create_with_security_group).

3. If create_with_security_group is True and an existing security group is specifed for security_group (as documented - "An existing security group with which to associate the network interface.") the module fails for me with Error creating or updating network interface nicName - Parameter 'FlowLog.target_resource_id' can not be None.. This exception is thrown in create_or_update_nic(self, nic) and, with _debug set to True, I can see in syslog it does find the existing NSG so I am little lost as to the cause of this fault.

ISSUE TYPE

• Bug Report

COMPONENT NAME

azure_rm_networkinterface

ANSIBLE VERSION

ansible [core 2.11.1] 
  config file = <redacted>/ansible.cfg
  configured module search path = ['<redacted>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = <redacted>/ansible-venv/lib/python3.8/site-packages/ansible
  ansible collection location = <redacted>/.ansible/collections:/usr/share/ansible/collections
  executable location = <redacted>/ansible-venv/bin/ansible
  python version = 3.8.3 (default, Jul  2 2020, 16:21:59) [GCC 7.3.0]
  jinja version = 3.0.1
  libyaml = True

COLLECTION VERSION

Collection         Version
------------------ -------
azure.azcollection 1.7.0  

CONFIGURATION

DISPLAY_SKIPPED_HOSTS(<redacted>/ansible.cfg) = False

OS / ENVIRONMENT

CentOS 7.8.2003

STEPS TO REPRODUCE

test_playbook.yml:

- name: Provision Resources
  hosts: localhost
  gather_facts: no
  tasks:
    - name: Network interface
      azure.azcollection.azure_rm_networkinterface:
        name: existingNIC
        ip_configurations:
          - name: ipconfig1
            primary: True
            private_ip_address: 10.0.0.1
            private_ip_allocation_method: Static
        location: westeurope
        resource_group: myRG
        create_with_security_group: True
        subnet_name: mySN
        security_group:
           name: myNSG
           resource_group: myNetRg
        virtual_network:
            name: myVNET
            resource_group: myNetRg

inventory (as inventory.azure_rm.yml):

plugin: azure_rm

auth_source: cli

keyed_groups:
  - key: resource_group
    prefix: rg

ansible-playbook -i inventory.azure_rm.yml test_playbook.yml

EXPECTED RESULTS

Play succeeds.

ACTUAL RESULTS

The full traceback is:
  File "/tmp/ansible_azure.azcollection.azure_rm_networkinterface_payload_mlljrgib/ansible_azure.azcollection.azure_rm_networkinterface_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_netwo
rkinterface.py", line 807, in create_or_update_nic
  File "<redacted>/ansible-venv/lib/python3.8/site-packages/azure/mgmt/network/v2020_06_01/operations/_network_interfaces_operations.py", line 269, in create_or_update
    raw_result = self._create_or_update_initial(
  File "<redacted>/ansible-venv/lib/python3.8/site-packages/azure/mgmt/network/v2020_06_01/operations/_network_interfaces_operations.py", line 220, in _create_or_update_initial
    body_content = self._serialize.body(parameters, 'NetworkInterface')
  File "<redacted>/ansible-venv/lib/python3.8/site-packages/msrest/serialization.py", line 578, in body
    raise errors[0]
  File "<redacted>/ansible-venv/lib/python3.8/site-packages/msrest/serialization.py", line 220, in validate
    Serializer.validate(value, debug_name, **self._validation.get(attr_name, {}))
  File "<redacted>/ansible-venv/lib/python3.8/site-packages/msrest/serialization.py", line 661, in validate
    raise ValidationError("required", name, True)
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "ad_user": null,
            "adfs_authority_url": null,
            "api_profile": "latest",
            "append_tags": true,
            "auth_source": "auto",
            "cert_validation_mode": null,
            "client_id": null,
            "cloud_environment": "AzureCloud",
            "create_with_security_group": true,
            "dns_servers": null,
            "enable_accelerated_networking": false,
            "enable_ip_forwarding": false,
            "ip_configurations": [
                {
                    "application_security_groups": null,
                    "load_balancer_backend_address_pools": null,
                    "name": "ipconfig1",
                    "primary": true,
                    "private_ip_address": "10.0.0.1",
                    "private_ip_allocation_method": "Static",
                    "public_ip_address_name": null,
                    "public_ip_allocation_method": "Dynamic"
                }
            ],
            "location": "westeurope",
            "log_mode": null,
            "log_path": null,
            "name": "existingNIC",
            "open_ports": null,
            "os_type": "Linux",
            "password": null,
            "private_ip_address": null,
            "private_ip_allocation_method": "Dynamic",
            "profile": null,
            "public_ip": true,
            "public_ip_address_name": null,
            "public_ip_allocation_method": "Dynamic",
            "resource_group": "myRG",
            "secret": null,
            "security_group": {
                "name": "myNSG",
                "resource_group": "myNetRg",
                "subscription_id": "<redacted>"
            },
            "state": "present",
            "subnet_name": "mySN",
            "subscription_id": null,
            "tags": null,
            "tenant": null,
            "virtual_network": {
                "name": "myVNET",
                "resource_group": "myNetRg",
                "subscription_id": "<redacted>"
            }
        }
    },
    "msg": "Error creating or updating network interface existingNIC - Parameter 'FlowLog.target_resource_id' can not be None."
}

[Fred-sun]

@LHurst-JM I have executed the following YAML locally and have not encountered the problem you mentioned? Can you try it again? Thank you very much!

- name: Create virtual network
  azure_rm_virtualnetwork:
      resource_group: "{{ resource_group }}"
      name: "tn{{ rpfx }}"
      address_prefixes: ["10.10.0.0/16", "fdae:f296:2787::/48"]
  register: vn

- name: Add subnet
  azure_rm_subnet:
      resource_group: "{{ resource_group }}"
      name: "tn{{ rpfx }}"
      address_prefixes_cidr: ["10.10.0.0/24", "fdae:f296:2787::/64"]
      virtual_network: "tn{{ rpfx }}"

- name: Add rules on existing security group
  azure_rm_securitygroup:
      resource_group: "{{ resource_group }}"
      name: testsecurity
      rules:
          - name: AllowSSH
            protocol: Tcp
            source_address_prefix: 174.108.158.0/24
            destination_port_range: 22
            access: Allow
            priority: 101
          - name: AllowSSHFromHome
            protocol: Tcp
            source_address_prefix: 174.109.158.0/24
            destination_port_range: 22-23
            priority: 102
          - name: AllowHTTPandHTTPS
            protocol: Tcp
            source_address_prefix: 174.109.158.0/24
            destination_port_range:
              - 80
              - 443
            priority: 103

- name: Create most simple NIC with virtual_network resource_group
  azure_rm_networkinterface:
      resource_group: "{{ resource_group }}"
      name: "tn{{ rpfx }}"
      ip_configurations:
        - name: ipconfig1
          primary: True
          private_ip_address: 10.10.0.11
          private_ip_allocation_method: Static
      location: eastus
      create_with_security_group: True
      security_group:
        resource_group: "{{ resource_group }}"
        name: testsecurity
      virtual_network:
        name: "tn{{ rpfx }}"
        resource_group: "{{ resource_group }}"
      subnet: "tn{{ rpfx }}"****

[LHurst-JM]

I'm still getting the same my end with my yaml:

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error creating or updating network interface existingNIC - Parameter 'FlowLog.target_resource_id' can not be None."}

I am trying to introduce Ansible to an existing infrastructure - so in my case the NIC, NSG and VNET already exist in Azure before I ran Ansible for the first time. At the moment it looks like TerraForm might be better for doing this...

[paultaiton]

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error creating or updating network interface existingNIC - Parameter 'FlowLog.target_resource_id' can not be None."}

I have also encountered this in my environment. It looks like there's a problem with the process that ultimately ends in calling the msrest Serializer class. If the security group has ever been set up to record Flow Logs, then the 'flow_logs' property will exist in the object returned by network_client.network_security_groups.list("rg-name"), but will be None regardless of the state in Azure. So when that object is sent to the azure_rm_common 'serialize_obj' method, and ultimately to the msrest serializer, it fails because it's expected to not be None.

I have tried using the most recent azure-mgmt-network package and Network API version, but I get the same result from NetworkManagementClient network_security_groups.list(rg-name) regardless.

I have not had the time to troubleshoot beyond that. At some point in the future if I get time I'll try sending the updated versions' through an updated msrest serializer to see if they at least accounted for the problem to get it to work. That would be a bad workaround though because then the Ansible module would have no accounting for Flow Logs, which I think we would want. I think ultimately this is a bug with the Azure python SDK, and don't see anything wrong with the azcollection code, just how the SDK handles it in this case.

[Fred-sun]

@LHurst-JM @paultaiton Can you share your test script? I will retry it! Thank you very much!

[paultaiton]

@LHurst-JM @paultaiton Can you share your test script? I will retry it! Thank you very much!

Howdy @Fred-sun Your test script is fine, but you have to set up flow logs for the NSG, which is currently not supported by azcollection, so I'd recommend just using Azure Portal. After that run securitygroup_info against it and you should get the error.

In my environment we have an Azure policy that automatically configures flow logs for any NSG, so we cant use securitygroup_info at all.

[LHurst-JM]

My test script is what I included in the original report (just changed the names - everything else is as in my test case) - I am trying to bring an existing infrastructure that was manually created under Ansible's control and I fell at the first hurdle, which was my VM's NIC.

I cannot confirm if there are flow logs for the NSG, I do not have visibility of that in our coporate infrastructure but I imagine that security do have them setup (also probably via policy enforcement like @paultaiton).

We seem to be ignoring my first 2 points too, that it is unclear from the docs that not setting 'create_with_security_group' to True will actually remove NSGs from NICs, even if security_group is set to the existing security groups - reading the docs I expected it to not touch the existing NSG configuration in this case. The parameter name (create_with_security_group ) and the associated documentation suggests that it only does anything on creating a new NIC and then only creates a new security group - nothing in the docs suggests not setting that to True will delete the existing NIC NSG configuration or that it controls whether security_group is applied if the NIC already exists, which seems to be the case.

[Fred-sun]

@LHurst-JM Could you please provide a complete Playbook and its implementation process? This goes a long way towards solving the problem! Thank you very much!

[Fred-sun]

@LHurst-JM Upgrading azure mgmt-network to the new SDK will fix this, fixex by #729! Thank you very much!