Bug Report - Linking 'privatelink.monitor.azure.com' Private DNS Zone Breaks Log Analytics Querying When AMPLS Not Used

[mbrat2005]

Describe the bug ALZ deployments currently create and link a 'privatelink.monitor.azure.com' Private DNS Zone to the hub VNET. Because of how Azure Monitor implements Private Endpoints (via an AMPLS), this default configuration breaks access to services such as Log Analytics queries from within VNET or clients with DNS queries being forwarded to this zone.

Azure Monitor is designed so DNS queries to Azure Monitor endpoints always return a privatelink CNAME with the response; most other Azure services implement Private Endpoints by returning a privatelink CNAME only when a Private Endpoint has been added to the resource. DNS clients prioritize the CNAME record in the response and attempt to recursively resolve the name (such as 'api.privatelink.monitor.azure.com') via the linked Private DNS Zone. The zone has no records, so it responds that the name does not exist. See: https://docs.microsoft.com/en-us/azure/azure-monitor/logs/private-link-security

With other services, linking the Private Endpoint Private DNS zones is not an issue, but this zone causes unexpected behavior which is frustrating to customers and difficult to troubleshoot. Unless ALZ also implements an AMPLS, as proposed in #967, this DNS zone should either not be linked to the hub VNET or not created altogether.

Steps to reproduce

1. Deploy ALZ (Contoso used)
2. [optional] Configure VNETs to use custom DNS IPs
3. [optional] On the custom DNS servers, configure a server-level forwarder or conditional forwarder (for zone 'privatelink.monitor.azure.com') to 168.63.129.16
4. Attempt to access Log Analytics data via the Portal or SDK

   PS C:\> Invoke-AzOperationalInsightsQuery -WorkspaceId '...' -Query "AzureDiagnostics"
   >>> Invoke-AzOperationalInsightsQuery : An error occurred while sending the request. <<<

Generic Steps: 1. Create a 'privatelink.monitor.azure.com' Private DNS Zone 2. Link the DNS zone to a VNET 3. Attempt to query Log Analytics from a VM within the VNET using Azure-provided DNS/168.63.129.16

Screenshots Packet capture of DNS query when Invoke-AzOperationalInsightsQuery is executed:

Error in PowerShell from a client with DNS queries forwarded to the Private DNS Zone:

Error in Portal from a client with DNS queries forwarded to the Private DNS Zone:

[jfaurskov]

Thank you @mbrat2005 for raising awareness on this. I'll look into it and get back to you soonest.

[jtracey93]

Trigger ADO Sync 1

[jtracey93]

Trigger ADO Sync 2

[kamilzzz]

Are there any updates? This should be at least mentioned in the docs as are the other Private DNS Zone issues (https://github.com/Azure/Enterprise-Scale/wiki/ALZ-Policies-FAQ#private-dns-zone-issues), ideally with scenarios it breaks.

Documentating scenarios may not be that obvious as I think with the new AMA agent this is even more messy. For example, in our case enabling Azure Monitor private link zones via TF ALZ module (it creates zones for monitor.azure.com, ods.opinsights.azure.com and oms.opinsights.azure.com) broken following scenarios:

• AKS clusters with Container Insights integration enabled are able to send data to LAW but they cannot access global.handler.control.monitor.azure.com and some of the pods are constantly restarting
• Log Querying is completely broken when using DNS forwarder with these privatelink zones linked
• Azure Managed Prometheus integration for AKS is completely broken and no metrics are published to Azure Monitor Workspace at all, in the logs we can see this is due to the global.handler.control.monitor.azure.com not being resolved correctly

The main issue from my perspective is that some Azure Monitor/Log Analytics DNS entries always return privatelink zone in the chain while some others are not. So for now entire Azure Monitor/Log Analytics stack is partially working and partially not.