azure_rm_deployment fails if no write permissions to the resource group are present

[sleepy-manul]

SUMMARY

I am using a service principal with restricted access. Using the Azure CLI (az deployment group create ...) I can successfully deploy my templates. The same does not work with azure_rm_deployment, most likely because it tries to write to the RG (to update tags?) in every case. Since I lack this permission, deployment with azure_rm_deployment fails:

Please note that this is very likely not an authentication problem; the fact that the Azure error message (see below) correctly shows the object guid of my service principal makes it very likely that authentication was successful, but that azure_rm_deployment might have sent a request which I am not allowed to execute with my privileges.

ISSUE TYPE

• Bug Report

COMPONENT NAME

azure_rm_deployment

ANSIBLE VERSION

ansible 2.10.1
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/nfs/my_username/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.8 (default, Apr  2 2020, 13:34:55) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

CONFIGURATION

INVENTORY_ENABLED(/etc/ansible/ansible.cfg) = ['host_list', 'virtualbox', 'yaml', 'constructed', 'azure_rm']

OS / ENVIRONMENT

Red Hat Enterprise Linux 7.9, Azure VM (linux-x64), all Updates installed.

STEPS TO REPRODUCE

Prepare a service principal that is allowed to deploy into a resource group, but not allowed to change the properties of the resource group itself (only read permissions on the RG). Then run the deployment with az CLI (should work) and then with azure_rm_deployment (expected not to work)

    - name: Deploy etcd/control plane load balancer and DNS entries
      azure_rm_deployment:
        tenant: "{{ tenant | mandatory }}"
        subscription_id: "{{ subscription_id | mandatory }}"
        resource_group: "{{ resource_group | mandatory }}"
        client_id: "my_service_principal_client_id"
        secret: 'my_service_principal_secret'
        name: "{{ deployment_name | mandatory }}"
        template: "{{ lookup('file', 'files/arm/tms-arm-template.json') }}"
        deployment_mode: incremental
        location: "{{ location | default('westeurope') }}"
        parameters:
          environmentName:
            value: "{{ environment_name | mandatory }}"

EXPECTED RESULTS

azure_rm_deployment should not try to write / update the RG if there is nothing to update

ACTUAL RESULTS

Azure shows a permission error and does not execute the deployment.

The most likely place where the problem occurs is https://github.com/ansible-collections/azure/blob/dev/plugins/modules/azure_rm_deployment.py#L548 (you can see there is always a call to self.rm_client.resource_groups.create_or_update, even though it is not necessary if the RG already exists and already has the desired configuration).

The full traceback is:
  File "/tmp/ansible_azure_rm_deployment_payload_c2a_o81z/ansible_azure_rm_deployment_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_deployment.py", line 548, in deploy_template
  File "/usr/local/lib/python3.6/site-packages/azure/mgmt/resource/resources/v2017_05_10/operations/resource_groups_operations.py", line 150, in create_or_update
    raise exp
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "ad_user": null,
            "adfs_authority_url": null,
            "api_profile": "latest",
            "append_tags": true,
            "auth_source": "auto",
            "cert_validation_mode": null,
            "client_id": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "cloud_environment": "AzureCloud",
            "deployment_mode": "incremental",
            "location": "westeurope",
            "name": "ansible_prod-test",
            "parameters": {
(omitted because not helpful in diagnostics)
            },
            "parameters_link": null,
            "password": null,
            "profile": null,
            "resource_group": "the_name_of_existing_resource_group",
            "secret": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "state": "present",
            "subscription_id": "id_of_my_azure_subscription",
            "tags": null,
            "template": "
(omitted because not helpful in diagnostics)
            },
,
            "template_link": null,
            "tenant": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "wait_for_deployment_completion": true,
            "wait_for_deployment_polling_period": 10
        }
    },
    "msg": "Resource group create_or_update failed with status code: 403 and message: The client 'guid_of_my_service_principal' with object id 'object_id_of_my_service_principal' does not have authorization to perform action 'Microsoft.Resources/subscriptions/resourcegroups/write' over scope '/subscriptions/guid_of_my_subscription/resourcegroups/name_of_my_resource_group' or the scope is invalid. If access was recently granted, please refresh your credentials."
}

[Fred-sun]

@stupiddog1979 Thank you for reporting this issue, we will review it as soon as possible, thank you!

[fepere]

Hello, actually the azure_rm_deployment is not working at all, I try to simply create a 'resource group' but is not working, it´s failing..

[johnpetersjr]

I don't know if this is the issue others are having, but I was able to solve my issue --

We have Azure Policy in place that requires certain tags to be present on resource groups. In my environment, I already had the resource group present, tagged appropriately, inside of which I wanted to deploy some resources using the azure_rm_deployment module. My experience is that the module tries to remove the tags from the resource_group provided in the module if those tags aren't present in the tags: section inside the module, which our Azure Policy forbids -- RG's must have these tags.

So -- I added the required tags in the module, (even though they are already present on the resource group) and that got me past the issue I was having. Hope that help someone else out there.

  azure_rm_deployment:
    resource_group: my-existing-resource-group
    name: some-new-arm-template-name
    state: present
    location: earth
    tags:
      required_tag1: yes_please
      required_tag2: yes_please
      required_tag3: yes_please

[sleepy-manul]

Hi @Fred-sun , I know this ticket has only been open for about 2.5 years, but do you see any possibility that you will be able to address it soon?

[Fred-sun]

@sleepy-manul I'm sorry for ignoring this problem. I will arrange time to deal with it as soon as possible. I'm terribly sorry!

[Fred-sun]

@sleepy-manul What resources do you want to deploy through this module? Could you please provide your use case and error log? This is more convenient to analyze your problems, thank you!

My playbook like below: --- demploy a new virtualnetwork 
- name: Create Azure Deploy
  azure_rm_deployment:
    resource_group: "{{ resource_group }}"
    location: "eastus"
    template: "{{ lookup('file', 'template.json') }}"
  register: output

[sleepy-manul]

Hi @Fred-sun , thank you (honestly) for taking the time. I really appreciate it.

The problem happens on every deployment. Even something as simple as a VM, a basic PostgreSQL server, or anything triggers the problem. Over time, I found a workaround. Right now, my deployments work again as soon as I comment out (=disable) the following lines from the source code:

    try:
        self.rm_client.resource_groups.create_or_update(self.resource_group, params)
    except Exception as exc:
        self.fail("Resource group create_or_update failed with status code: %s and message: %s" %
                  (exc.status_code, exc.message))

The link to the current version is: https://github.com/ansible-collections/azure/blob/dev/plugins/modules/azure_rm_deployment.py#L541

However, using this workaround is problematic since we must manually patch the Azure Collections Ansible Collection every time we want to upgrade to a new version.

The most likely cause why this problem occurs in our environment is:

• We are working in a strictly controlled environment (corporate / enterprise politics), and we get very few permissions on the Azure cloud.
• In our example, we are allowed to deploy things like VMs, databases or anything else inside an existing (!) resource group. Still, we are forbidden/locked from editing the resource group itself. This even includes changing things like Azure tags assigned to a resource group.

Specifically, we lack the permission (Azure Subscription)/resourceGroups/(Name of the Resouce Group)/write .

However, in the program code above, we see that the module calls the Azure SDK function resource_groups.create_or_update(). In our situation, this is unnecessary (because the resource group already exists and has everything we need). Additionally, it causes an error message from Azure (that the permission mentioned above is missing).

I believe the reason why the Python code tries this is to ensure that the resource group exists and that it is ready for deployment. However, I think the code should better check for the existence of the RG, which is something we would be allowed to do.

[Fred-sun]

@sleepy-manul If the az command can be used to execute the command successfully。 Maybe your Service principal have no permission. You are advised to use az login and specify auth_source: cli. Thanks!

like:
  azure_rm_deployment:
    resource_group: my-existing-resource-group
    name: some-new-arm-template-name
    state: present
    location: earth
    auth_source: cli
    tags:
      required_tag1: yes_please
      required_tag2: yes_please
      required_tag3: yes_please