Appinsights hidden-link tags are beeing added and removed over and over

[aersam]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.1.9

AzureRM Provider Version

3.3.0

Affected Resource(s)/Data Source(s)

azurerm_linux_function_app

Terraform Configuration Files

resource "azurerm_linux_function_app" "powerbi_stream" {
  name = "func-powerbistream-${local.project}-${var.environment}-${var.region}-001"
  service_plan_id = azurerm_service_plan.dataplatform.id
  location = azurerm_storage_account.dataplatform.location
  resource_group_name = azurerm_resource_group.dataplatform.name
  tags = local.tags

  identity {
    type = "SystemAssigned"
  }

  storage_account_name = azurerm_storage_account.dataplatform.name
  storage_account_access_key = azurerm_storage_account.dataplatform.primary_access_key   
  site_config {
    ftps_state = "Disabled"
    http2_enabled = true
    always_on = false
    application_insights_key = azurerm_application_insights.dataplatform.instrumentation_key
    application_insights_connection_string = azurerm_application_insights.dataplatform.connection_string
  }

  app_settings = {
    "TARGET_STORAGE_ACCOUNT_URL": "https://${azurerm_storage_account.dataplatform.name}.blob.core.windows.net/"
    "TARGET_CONTAINER_NAME": azurerm_storage_data_lake_gen2_filesystem.dataplatform_lake_raw.name
    "AzureWebJobsSecretStorageType": "keyvault"
    "AzureWebJobsSecretStorageKeyVaultUri": azurerm_key_vault.powerbi_stream_keys.vault_uri    
    "WEBSITE_RUN_FROM_PACKAGE": ""
    "WEBSITE_ENABLE_SYNC_UPDATE_SITE": "false"
  }
  
  https_only = true
}

Debug Output/Panic Output

~ tags                              = {
          - "hidden-link: /app-insights-instrumentation-key" = "SOMEKEY" -> null
          - "hidden-link: /app-insights-resource-id"         = "/subscriptions/SUBID/resourceGroups/RGNAME/providers/microsoft.insights/components/APPINAME" -> null
            # (3 unchanged elements hidden)
        }

Expected Behaviour

Terraform seems to add these hidden-link tags implicity when linking app insights. However it attempts to remove them on next run

Actual Behaviour

Terraform should either not add these hidden-link tags (however I think they are needed for app insights) or then recognize them and ignore them

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[jennings]

These "hidden-link" and "hidden-related" tags just started appearing for us in the last day for function apps and storage accounts that we haven't changed. I found #13829 from October 2021 for this, and so I thought maybe we had changed something. But seeing this ticket open today makes me think others are suddenly seeing this as well.

I'm using azurerm provider version 3.0.2, and these tags didn't appear until well after I upgraded to that version. We're using Terraform Cloud to execute runs, with Terraform version 1.1.7 selected.

  # module.<redacted>.azurerm_storage_account.this will be updated in-place
  ~ resource "azurerm_storage_account" "this" {
      <redacted>
      ~ tags                              = {
          - "hidden-related:/providers/Microsoft.Web/sites/<redacted>" = "empty" -> null
        }
        # (26 unchanged attributes hidden)


        # (5 unchanged blocks hidden)
    }

  # module.<redacted>.azurerm_windows_function_app.<redacted> will be updated in-place
  ~ resource "azurerm_windows_function_app" "<redacted>" {
      <redacted>
      ~ tags                              = {
          - "hidden-link: /app-insights-instrumentation-key" = "<redacted>" -> null
          - "hidden-link: /app-insights-resource-id"         = "/subscriptions/<redacted>/resourceGroups/<redacted>/providers/microsoft.insights/components/<redacted>" -> null
            # (2 unchanged elements hidden)
        }
        # (25 unchanged attributes hidden)


        # (2 unchanged blocks hidden)
    }

[tombuildsstuff]

hi @aersamkull

Thanks for opening this issue.

The hidden-link tag is used to link Application Insights to a particular resource, so that's expected if the other resource is connected to Application Insights (and would be something you'd need to match in your Terraform Configuration, else when the tag is removed the association will be removed) - unfortunately this is the only means that exists of associating Application Insights to other resources at present.

The hidden-related tag however appears to be a breaking change made by the Function App API's - I'll ask @jackofallops to take a look into this, but this would likely need to be worked around in the same manner (by adding this to the Terraform Configuration).

Whilst we'd generally ask that configuration questions are opened on the Community Discuss forum - since the hidden-related tag appears to be an upstream breaking change I'll hold off redirecting this issue for the moment, since it appears this breaking behavioural change to the Azure API may need documenting.

Thanks!

[aersam]

Ok I have to go through some docu, maybe I missed something. I am quite sure however that this started appearing after upgrading to v3 of azurerm, I'll check the logs

[jackofallops]

Hi @aersamkull / @jennings - Can you share the locations you're deploying into? I just tried to reproduce this in West Europe and tags are behaving as expected (i.e. not returning the hidden items in the API response)

Thanks!

[aersam]

Location is Switzerland North

[aersam]

I can check tomorrow if its really terraform adding the tags or if it happens on starting the function or even on deployment using Azure DevOps

[jackofallops]

I can check tomorrow if its really terraform adding the tags or if it happens on starting the function or even on deployment using Azure DevOps

Thanks @aersamkull - I've just tried in Switzerland North, and I'm not seeing the hidden tags being returned on the resource creation.

My test doesn't deploy to the app, but does connect App Insights:

I'm on UK time, so it's quite late right now, but I'll check back tomorrow 👍

[jennings]

@jackofallops I'm in West US. I just checked the prod and test version of a function app we have. The test version has the hidden-link tags, the prod version doesn't. The azurerm_windows_function_app that defines them is in a module being used twice, so their configuration is exactly the same.

I threw both state JSON objects into kdiff3, and the only differences between the two are:

• A couple app_settings values
• identity.principal_id
• name
• site_config.application_insights_connection_string
• site_config.application_insights_key
• site_config.elastic_instance_minimum
• site_config.scm_type
• site_credential.password
• And the test version has the two hidden-link tags, in addition to the two tags I'm assigning in Terraform config

[jackofallops]

Thanks for the additional info @jennings

The provider doesn't set those tags, nor does it expect them to be there if not explicitly specified in the tags block, so I can only assume that something is modifying them outside of your infrastructure code, or as @tombuildsstuff mentioned above, something has changed in the service. For the time being, you can add them to prevent Terraform making changes, or

You included the site_config.scm_type property in the diff, we don't modify that directly, it's a Computed attribute in the provider, which suggests that your two apps are deployed to differently? (thinking this could possibly be the source, e.g. ADO, as @aersamkull is using?) Also, if they were created in Azure on either side of a change in the service, they could have differing configuration from the same API, unfortunately. (I'm guessing that Prod is longer lived here?)

Thanks again for the continued info to get to the bottom of this. It's looking like an Azure Service side change to me, so I think we need to find out if it's intentional to know what steps to take next. I'll try and reach out to the Service Team at Azure for clarification.

[rvava]

Hello,

I also have the issue on West Europe for a few days now. I'm using azurerm_function_app in v2.99 (which I know is deprecated, I will upgrade to v3 next week).

While waiting for something else, here is a little work around : you can add the lifecycle -> ignore_changes block for Terraform to ignore the changes on specific part.

resource "azurerm_function_app" "function" {
  
  [...]

  lifecycle {
    ignore_changes = [
      tags["hidden-link: /app-insights-instrumentation-key"],
      tags["hidden-link: /app-insights-resource-id"]
    ]
  }
}

And no more changes detected!

Hope this help.

[scottmckenzie]

I've added the "hidden-link" tags to my ignore_changes block. This does stop terraform from updating the tags. However, when I run terraform plan, the value of the tags is output to the plan. This means my AppInsights instrumentation key is now visible to the whole world through the githubs actions runs in my repo. Here's an example:

Terraform will perform the following actions:

  # azurerm_linux_function_app.app will be updated in-place
  ~ resource "azurerm_linux_function_app" "app" {
      ~ app_settings                      = {
          ~ "WEBSITE_RUN_FROM_PACKAGE"               = (sensitive)
            # (3 unchanged elements hidden)
        }
        id                                = "/subscriptions/ffffffff-ffff-ffff-ffff-ffffffffffff/resourceGroups/rg-my-rg/providers/Microsoft.Web/sites/func-my-funcapp"
        name                              = "func-my-funcapp"
        tags                              = {
            "hidden-link: /app-insights-conn-string"         = "InstrumentationKey=ffffffff-ffff-ffff-ffff-ffffffffffff;IngestionEndpoint=https://australiaeast-0.in.applicationinsights.azure.com/;LiveEndpoint=https://australiaeast.livediagnostics.monitor.azure.com/"
            "hidden-link: /app-insights-instrumentation-key" = "ffffffff-ffff-ffff-ffff-ffffffffffff"
            "hidden-link: /app-insights-resource-id"         = "/subscriptions/ffffffff-ffff-ffff-ffff-ffffffffffff/resourceGroups/rg-my-rg/providers/microsoft.insights/components/appi-my-funcapp"
        }
        # (24 unchanged attributes hidden)

This doesn't happen on apply though:

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # azurerm_linux_function_app.app will be updated in-place
  ~ resource "azurerm_linux_function_app" "app" {
      ~ app_settings                      = {
          ~ "WEBSITE_RUN_FROM_PACKAGE"      = (sensitive)
            # (3 unchanged elements hidden)
        }
        id                                = "/subscriptions/***/resourceGroups/rg-my-rg/providers/Microsoft.Web/sites/func-my-funcapp"
        name                              = "func-my-funcapp"
        tags                              = {}

[aersam]

Hello,

I also have the issue on West Europe for a few days now. I'm using azurerm_function_app in v2.99 (which I know is deprecated, I will upgrade to v3 next week).

While waiting for something else, here is a little work around : you can add the lifecycle -> ignore_changes block for Terraform to ignore the changes on specific part.

resource "azurerm_function_app" "function" {
  
  [...]

  lifecycle {
    ignore_changes = [
      tags["hidden-link: /app-insights-instrumentation-key"],
      tags["hidden-link: /app-insights-resource-id"]
    ]
  }
}

And no more changes detected!

Hope this help.

I used that workaround, too. Works for me but it should definitely get fixed

[scottmckenzie]

Has anyone successfully used the tag workaround on the azurerm_linux_function_app resource in v3 of hashicorp/azurerm? i.e. the resource this issue was logged against.

Saying it works for azurerm_function_app in v2 is not helpful, nor relevant to this issue.

[William-LP]

Has anyone successfully used the tag workaround on the azurerm_linux_function_app resource in v3 of hashicorp/azurerm? i.e. the resource this issue was logged against.

Saying it works for azurerm_function_app in v2 is not helpful, nor relevant to this issue.

I'm running

    azurerm = {
      version = ">= 3.6.0"
    }

and

resource "azurerm_linux_function_app" "my_func" {
...
  lifecycle {
    ignore_changes = [
      tags["hidden-link: /app-insights-instrumentation-key"],
      tags["hidden-link: /app-insights-resource-id"],
      tags["hidden-link: /app-insights-conn-string"]
    ]
  }
...
}

The workaround works as expected.

[angelbulas]

Hello,

I also have the issue on West Europe for a few days now. I'm using azurerm_function_app in v2.99 (which I know is deprecated, I will upgrade to v3 next week).

While waiting for something else, here is a little work around : you can add the lifecycle -> ignore_changes block for Terraform to ignore the changes on specific part.

resource "azurerm_function_app" "function" {
  
  [...]

  lifecycle {
    ignore_changes = [
      tags["hidden-link: /app-insights-instrumentation-key"],
      tags["hidden-link: /app-insights-resource-id"]
    ]
  }
}

And no more changes detected!

Hope this help.

Works perfect, thank you

[GK-Matias]

There is an optional argument for the function app resource: enable_builtin_logging that defaults to true. For me these hidden-links seems to somehow be connected with this setting. When I disabled the built in logging these settings seemed to disappear. Maybe this can help someone?

Reference: https://www.ericksegaar.com/2020/09/15/terraform-defaults-slowly-filling-azure-function-storage-account/

[Tbohunek]

Indeed the workaround is what Terraform azurerm should do by default... If even Azure needs to show these on the resource json ☹️

[eltimmo]

I've been having issues with for months, I ended on this thread as I wanted to understand exactly what these are for. One work around is just to merge them in with your tags.

locals {

  function_app_dynamic_tags = tomap({
    "hidden-link: /app-insights-resource-id"         = replace(azurerm_application_insights.application_insights.id, "Microsoft.Insights", "microsoft.insights")
    "hidden-link: /app-insights-conn-string"         = azurerm_application_insights.application_insights.connection_string
    "hidden-link: /app-insights-instrumentation-key" = azurerm_application_insights.application_insights.instrumentation_key
  })

}

resource "azurerm_linux_function_app" "linux_function_app" {

 tags  = merge(var.tags, local.function_app_dynamic_tags)

}

[vijaypathak26]

I have had similar issue even after removing the default tags, so i've add same hidden-link to match to exact subscription

under the module

1. Main.tf locals { tags = merge(var.tags, { "hidden-link:${var.ainId}":"Resource" # var.ainId =application_insights_id i.e. "/subscriptions/{subscriptionID}/resourceGroups/{resourcegpName}/providers/Microsoft.Insights/components/{application insight name}" } ) }

And then availability.tf tags = merge(local.tags, var.tags)

lifecycle { ignore_changes = tags["environment"], tags["CreatedOnDate"], tags["Environment"], tags["Creator"]] }

It worked for me !

[wbwlkr]

Hi @jackofallops,

I'm in West Europe, I just had the same issue and it was fixed with the workaround suggested above :

add the lifecycle -> ignore_changes block for Terraform to ignore the changes on specific part.

I was wondering if you know if that behavior will be the default one for Terraform azurerm soon ?

Thank you everyone

[jackofallops]

Hi @wbwlkr - I'm afraid this isn't a problem in the provider, but seems to be a bug in the service itself in some cases. The hidden tags are not supposed to be returned in the API response as they are intended for internal service use only, not for end-users (hence the hidden prefix 🙈 ). I'll try passing on this information to our contacts at Azure again to see if there's a broader issue being tracked / worked on.

[shoop]

I'm seeing effectively the same thing when setting the Azure Container Registry for a Linux container app:

      ~ tags                              = {
          - "hidden-link: acrResourceId" = jsonencode(
                {
                  - resourceId     = "/subscriptions/XXXX/resourceGroups/XXXX/providers/Microsoft.ContainerRegistry/registries/XXXX"
                  - subscriptionId = "XXXX"
                }
            ) -> null
        }

So it's not only for the AppInsights link that these get added.

[jackofallops]

Hi all

Just by way of update, we're in discussions with Azure on this. There's an implementation change that is being rolled out that is exposing these hidden tags. The Upstream tracking issue is https://github.com/Azure/Azure-Functions/issues/2243. We're taking a look at how best to deal with this in the provider, but also pushing for these to be filtered from the responses as they are an implementation detail that we feel should not be exposed to users. In the meantime, use of ignore_changes is advised where possible to prevent these unexpected tags causing a diff or problems that may occur by Terraform removing these unexpected items.

Thanks for your patience!

[fardarter]

There is an optional argument for the function app resource: enable_builtin_logging that defaults to true. For me these hidden-links seems to somehow be connected with this setting. When I disabled the built in logging these settings seemed to disappear. Maybe this can help someone?

Reference: https://www.ericksegaar.com/2020/09/15/terraform-defaults-slowly-filling-azure-function-storage-account/

This seemed to work for me over three runs. Attribute is builtin_logging_enabled = false on azurerm_linux_function_app.

[drdamour]

sure would be nice if that implementation detail could leverage the keyvault ref connection string

[philippbussche]

I am also seeing the issue of these hidden-link tags but even when I add them to my ignore_changes config the plan says they would still be removed. I wonder if this is maybe due to using a stone-age old version of Terraform being < 1.0 ?

[vivaladan]

@jackofallops the upstream issue you mentioned https://github.com/Azure/Azure-Functions/issues/2243 was last week marked as closed, with a won't fix on it. although i'm not clear how it intersects with what we are suffering from here.

[Tbohunek]

@vivaladan I think that issue got mixed up, it has nothing to do with this problem. Instrumentation key is available in Properties of AppInsights resource. This ticket is just about Terraform ignoring the hidden-link tags so users don't have to and don't accidentally overwrite it.

[tombuildsstuff]

@Tbohunek since these tags are what's used to link these two resources together, unfortunately we aren't planning to filter these out (or manage these for you automatically) - since doing so would unintentionally leave resources connected to Application Insights which has a cost implication.

Within Azure, resources are linked by having one of the resources reference the other (via PATCHing that resource) - this is different to other systems which tend to have an "association" resource linking the two (which has a full CRUD lifecycle). Whilst this can make sense architecturally, unfortunately this design choice ends up bleeding out to consumers of the API (as you're seeing) - and in this case the association between Application Insights and the Azure Function (or any other Azure Resource) is managed through the hidden-link tags.

Whilst I can understand the appeal for filtering these out, unfortunately this is the design of the Azure API and so we're somewhat limited in what we can do here. If we filtered these out (and silently added them back to the resource being deployed) - how would users remove said association between these two resources? Unfortunately if we did so then users wouldn't be aware they're unintentionally sending logs from that resource to App Insights, which will have cost implementations for them - and may become a problem.

Ultimately this comes down to the design of the Azure API - and it's worth calling out this is a problem for us in a number of areas - but unfortunately we can't filter these tags out without introducing both unintended costs for users (being unaware that these logs are being forwarded on) and (ultimately) a security issue, since we'd be masking that we're sending logs (which could contain anything, including PII) to an arbitrary Application Insights resource.

Whilst I appreciate this probably isn't the outcome you were looking for, since this is by design within the Azure API - unfortunately I'm going to have to close this issue out as working as intended at this point in time - and our recommendation is to add the missing hidden-link tag to link the Application Insights Component to the relevant Azure Resource.

Thanks!

[drdamour]

@tombuildsstuff

unintentionally leave resources connected to Application Insights which has a cost implication.

are we sure about this? as i understand it this simply drives some blade integration in portal, it does NOT cause data to go app insights or not. (with or without them my data ends up in app insights).