Azure
New-AzDataCollectionRule. does not create DCR endoints (logs& Metrics)
Description
Hello Contrary to ARM deployment the DCR logs and metrics endpoints are not generated when created a new DRC with PowerShell
Issue script & Debug output
New-AzDataCollectionRule -ResourceGroupName 'RGxxxxx' -Name 'DCR-ReproTest' -JsonFilePath '/home/azadm/New_DCR_AZPolicyComplianceDetails.json'
DEBUG: 11:45:31 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - GetAzureRMContextCommand begin processing with ParameterSet 'GetSingleContext'.
DEBUG: 11:45:31 AM - [ConfigManager] Got [False] from [DisplayBreakingChangeWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [CheckForUpgrade], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 11:45:31 AM - GetAzureRMContextCommand end processing.
DEBUG: [CmdletBeginProcessing]: Starting command
DEBUG: CmdletBeginProcessing:
DEBUG: CmdletProcessRecordStart:
DEBUG: CmdletGetPipeline:
DEBUG: CmdletBeforeAPICall:
DEBUG: URLCreated: /subscriptions/yyyyyy/resourceGroups/RGxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01
DEBUG: RequestCreated: /subscriptions/yyyyyy/resourceGroups/RGxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01
DEBUG: HeaderParametersAdded:
DEBUG: BodyContentSet:
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [DisableInstanceDiscovery], Module = [], Cmdlet = []. Returning default value [False].
DEBUG: ============================ HTTP REQUEST ============================
HTTP Method:
PUT
Absolute Uri:
https://management.azure.com/subscriptions/yyyy/resourceGroups/RGxxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01
Headers:
x-ms-unique-id : 2
x-ms-client-request-id :yyyyyyyy
CommandName : New-AzDataCollectionRule
FullCommandName : New-AzDataCollectionRule_CreateViaJsonFilePath
ParameterSetName : __AllParameterSets
User-Agent : AzurePowershell/v12.1.0,PSVersion/v7.4.3,Az.DataCollectionRule/5.2.1
Body:
{
"location": "westeurope",
"properties": {
"streamDeclarations": {
"Custom-Historical_AzPolicyComplianceDetails_CL": {
"columns": [
{
"name": "policyAssignmentId",
"type": "string"
},
{
"name": "policyDefinitionId",
"type": "string"
},
{
"name": "policyDefinitionReferenceId",
"type": "string"
},
{
"name": "policyDefinitionGroupNames",
"type": "string"
},
{
"name": "policyDefinitionAction",
"type": "string"
},
{
"name": "numberOfNonCompliantResources",
"type": "int"
},
{
"name": "numberOfCompliantResources",
"type": "int"
},
{
"name": "details",
"type": "dynamic"
}
]
}
},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "/subscriptions/xxxxxx/resourceGroups/rgxxxxxxx/providers/microsoft.operationalinsights/workspaces/policyworkspace",
"workspaceId": "xxxx",
"name": "myworkspace"
}
]
},
"dataFlows": [
{
"streams": [
"Custom-Historical_AzPolicyComplianceDetails_CL"
],
"destinations": [
"myworkspace"
],
"transformKql": "source\n| extend TimeGenerated = now()\n",
"outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
}
]
}
}
DEBUG: BeforeCall:
DEBUG: ============================ HTTP RESPONSE ============================
Status Code:
OK
Headers:
Cache-Control : no-cache
Pragma : no-cache
Vary : Accept-Encoding
x-ms-ratelimit-remaining-subscription-resource-requests: 149
Request-Context : appId=cid-v1:x
x-ms-correlation-request-id : x
x-ms-client-request-id : x
x-ms-routing-request-id : WESTEUROPE:xx
x-ms-request-id : xxxx
api-supported-versions : 2019-11-01-preview, 2021-04-01, 2021-09-01-preview, 2022-06-01, 2023-03-11, 2024-03-11
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-Content-Type-Options : nosniff
X-Cache : CONFIG_NOCACHE
X-MSEdge-Ref : Ref A: xxxx Ref B: xxxx Ref C: 2024-08-01T11:45:31Z
Date : Thu, 01 Aug 2024 11:45:33 GMT
Body:
{
"properties": {
"immutableId": "dcr-ewfwff3",
"streamDeclarations": {
"Custom-Historical_AzPolicyComplianceDetails_CL": {
"columns": [
{
"name": "policyAssignmentId",
"type": "string"
},
{
"name": "policyDefinitionId",
"type": "string"
},
{
"name": "policyDefinitionReferenceId",
"type": "string"
},
{
"name": "policyDefinitionGroupNames",
"type": "string"
},
{
"name": "policyDefinitionAction",
"type": "string"
},
{
"name": "numberOfNonCompliantResources",
"type": "int"
},
{
"name": "numberOfCompliantResources",
"type": "int"
},
{
"name": "details",
"type": "dynamic"
}
]
}
},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "/subscriptions/9yyyyy/resourceGroups/rg-int-dgs-lab-its-itinfra-1/providers/microsoft.operationalinsights/workspaces/policyworkspace",
"workspaceId": "zzzzzz",
"name": "myworkspace"
}
]
},
"dataFlows": [
{
"streams": [
"Custom-Historical_AzPolicyComplianceDetails_CL"
],
"destinations": [
"myworkspace"
],
"transformKql": "source\n| extend TimeGenerated = now()\n",
"outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
}
],
"provisioningState": "Succeeded"
},
"location": "westeurope",
"id": "/subscriptions/yyyyy/resourceGroups/RGxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest",
"name": "DCR-ReproTest",
"type": "Microsoft.Insights/dataCollectionRules",
"etag": "\"b6009126-0000-0d00-0000-66ab755d0000\"",
"systemData": {
"createdBy": "xxxxx",
"createdByType": "User",
"createdAt": "2024-08-01T11:45:31.8835297Z",
"lastModifiedBy": "xxxxx",
"lastModifiedByType": "User",
"lastModifiedAt": "2024-08-01T11:45:31.8835297Z"
}
}
DEBUG: ResponseCreated:
DEBUG: BeforeResponseDispatch:
DEBUG: Finally:
DEBUG: CmdletAfterAPICall:
DEBUG: [CmdletProcessRecordAsyncEnd]: Finish HTTP process
DEBUG: CmdletProcessRecordAsyncEnd:
DEBUG: CmdletProcessRecordEnd:
DEBUG: 11:45:33 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].
DataCollectionEndpointId :
DataFlow : {{
"streams": [ "Custom-Historical_AzPolicyComplianceDetails_CL" ],
"destinations": [ "myworkspace" ],
"transformKql": "source\n| extend TimeGenerated = now()\n",
"outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
}}
DataSourceDataImportEventHubConsumerGroup :
DataSourceDataImportEventHubName :
DataSourceDataImportEventHubStream :
DataSourceExtension :
DataSourceIisLog :
DataSourceLogFile :
DataSourcePerformanceCounter :
DataSourcePlatformTelemetry :
DataSourcePrometheusForwarder :
DataSourceSyslog :
DataSourceWindowsEventLog :
DataSourceWindowsFirewallLog :
Description :
DestinationAzureMonitorMetricName :
DestinationEventHub :
DestinationEventHubsDirect :
DestinationLogAnalytic : {{
"workspaceResourceId": "/subscriptions/xxxxxxx/resourceGroups/rgxxxxxx/providers/microsoft.operationalinsights/workspaces/policyworkspace",
"workspaceId": "4yyyyy",
"name": "myworkspace"
}}
DestinationMonitoringAccount :
DestinationStorageAccount :
DestinationStorageBlobsDirect :
DestinationStorageTablesDirect :
Etag : "b6009126-0000-0d00-0000-66ab755d0000"
Id : /subscriptions/9XXXXX/resourceGroups/RGXXXXXX1/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest
IdentityPrincipalId :
IdentityTenantId :
IdentityType :
IdentityUserAssignedIdentity : {
}
ImmutableId : dcr-2e40a7469fXXXXXX
Kind :
Location : westeurope
MetadataProvisionedBy :
MetadataProvisionedByResourceId :
Name : DCR-ReproTest
ProvisioningState : Succeeded
ResourceGroupName : RGXXXX
StreamDeclaration : {
"Custom-Historical_AzPolicyComplianceDetails_CL": {
"columns": [
{
"name": "policyAssignmentId",
"type": "string"
},
{
"name": "policyDefinitionId",
"type": "string"
},
{
"name": "policyDefinitionReferenceId",
"type": "string"
},
{
"name": "policyDefinitionGroupNames",
"type": "string"
},
{
"name": "policyDefinitionAction",
"type": "string"
},
{
"name": "numberOfNonCompliantResources",
"type": "int"
},
{
"name": "numberOfCompliantResources",
"type": "int"
},
{
"name": "details",
"type": "dynamic"
}
]
}
}
SystemDataCreatedAt : 8/1/2024 11:45:31 AM
SystemDataCreatedBy : xxxx
SystemDataCreatedByType : User
SystemDataLastModifiedAt : 8/1/2024 11:45:31 AM
SystemDataLastModifiedBy : xxx
SystemDataLastModifiedByType : User
Tag : {
}
Type : Microsoft.Insights/dataCollectionRules
DEBUG: AzureQoSEvent: Module: Az.Monitor:5.2.1; CommandName: New-AzDataCollectionRule; PSVersion: 7.4.3; IsSuccess: True; Duration: 00:00:02.5406602; SanitizeDuration: 00:00:00.0186410
Environment data
Name Value
---- -----
PSVersion 7.4.3
PSEdition Core
GitCommitId 7.4.3
OS CBL-Mariner/Linux
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
Module versions
ModuleType Version PreRelease Name ExportedCommands
---------- ------- ---------- ---- ----------------
Script 3.0.2 Az.Accounts {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script 8.0.0 Az.Compute {Add-AzImageDataDisk, Add-AzVhd, Add-AzVMAdditionalUnattendContent, Add-AzVMDataDisk…}
Script 7.8.0 Az.Network {Add-AzApplicationGatewayAuthenticationCertificate, Add-AzApplicationGatewayBackendAddressPool, Add-AzApplicationGatewayBackendHttpSetting, Add-AzApplicationGatewayBackendSetting…}
Script 7.1.0 Az.Resources {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Script 7.0.0 Az.Storage {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountManagementPolicyAction, Add-AzStorageAccountNetworkRule, Close-AzStorageFileHandle…}
Script 1.1.3 Az.Tools.Predictor {Disable-AzPredictor, Enable-AzPredictor, Open-AzPredictorSurvey, Send-AzPredictorRating}
Script 0.0.0.10 AzureAD.Standard.Preview {Add-AzureADApplicationOwner, Add-AzureADDeviceRegisteredOwner, Add-AzureADDeviceRegisteredUser, Add-AzureADDirectoryRoleMember…}
Script 0.9.3 AzurePSDrive
Error output
No response
Ran into this as well today, I'm pretty sure because it is still using the 2022-06-01 as seen in the trace, and needs to be bumped to the 2023-03-11 api version
https://learn.microsoft.com/en-us/rest/api/monitor/data-collection-rules/create?view=rest-monitor-2023-03-11&tabs=HTTP
Is there any way we can force the API version except going again to Invoke-RestMethod...
@ChristopheLux I made a custom one that uses the newer API version and I still didn't see the endpoint getting populated, so I'm not sure what's going on, I was going to open a ticket and report back.
EDIT: https://gist.github.com/JustinGrote/22c4963f7eb5af08399c26cbf60bc3ae
OK, I think I figured it out.
As of the API spec, there is an ingestion endpoint example where you have to specify the kind as "Direct", note there is a typo, there's an extraneous space in this.
https://learn.microsoft.com/en-us/rest/api/monitor/data-collection-rules/create?view=rest-monitor-2023-03-11&tabs=HTTP#create-or-update-data-collection-rule-with-embedded-ingestion-endpoints
Even though the specs for the Kind parameter say only Windows and Linux are supported values.
I updated my script to specify the kind as Direct, and now I got ingestionEndpoints populated.
"endpoints": {
"logsIngestion": "https://xxxx-westus3.logs.z1.ingest.monitor.azure.com",
"metricsIngestion": "https://xxx-westus3.metrics.z1.ingest.monitor.azure.com"
},
With some more experimentation in regards to Kind, by supplying invalid data to the API, I get back an error that says these are the actual valid values:
Direct,Linux,Windows,WorkspaceTransforms,AgentDirectToStore,AgentSettings,PlatformTelemetry
These appear to be undocumented with a quick google search other than Linux and Windows, these do come back via the 2022 API,
~~and a test of the 2022 API with Direct does populate the endpoints it seems (there's a significant delay, it's not immediate, some sort of provisioning delay)~~ EDIT: Later testing shows this is not the case
and the endpoints can ONLY be seen with the 2023 API.
@JustinGrote in the documentation for the PowerShell there is https://learn.microsoft.com/en-us/powershell/module/az.monitor/new-azdatacollectionrule?view=azps-12.1.0 the -Kind. Stupid me
@JustinGrote in the documentation for the PowerShell there is https://learn.microsoft.com/en-us/powershell/module/az.monitor/new-azdatacollectionrule?view=azps-12.1.0 the -Kind. Stupid me
yeah but in my initial testing it doesn't seem to populate endpoints unless the API version is 2023 for the PUT, I'm testing that now.
OK based on this testing with my custom cmdlet:
New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2023Direct' -ApiVersion '2023-03-11'
New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2022Direct' -ApiVersion '2022-06-01'
New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2022DirectReplace' -ApiVersion '2022-06-01'
New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2022DirectReplace' -ApiVersion '2023-03-11' #Overwrites previous
#Additional custom attempt of 2023 API but with Kind not specified at all.
Findings
- You must use 2023 API, no 2022 attempts caused the endpoints to populate
- You can use PUT or PATCH to update a 2022 API to 2023 and as long as the Kind is Direct, it will get the endpoint property
- You must query using 2023 to see the endpoints property, doesn't exist in 2022
- Leaving the kind property unspecified does NOT populate the endpoints even with 2023, it looks like you have to use Kind: Direct
Pretty annoying the DCR docs don't mention that Kind: Direct is required
So currently getting DCR endpoints populated is not possible until the API rev gets bumped on this command, you have to use my custom workaround script. I'll updated it and relink https://github.com/Azure/azure-powershell/issues/25727#issuecomment-2265860351
@isra-fel the DataCollectionRule.Autorest needs a bump to 2023-03-11 to resolve this issue.
Hi JustinGrote, may I know how to do the workaround? I am using "kind": "Direct" and facing the same issue and need to change the api version to '2023-03-11' so that the DCR ingestion endpoints are created. However, I have no idea how to update the api version. Could you provide any help please?
