azure-cli-extensions icon indicating copy to clipboard operation
azure-cli-extensions copied to clipboard

Published 20 hours ago • verified publisher icon Azure

Can not create data-collection rule with a custom stream

Open Joris-van-der-Wel opened this issue 11 months ago • 3 comments

Describe the bug

I am attempting to create a data collection rule in azure monitor, that is intended to be used with a custom application that calls the Logs Ingestion API. I have created a logs-analytics workspace containing a custom table. My data collection rule should be configured without any data sources, instead a custom stream for my table is declared.

This all works fine when I configure this via the Azure Portal GUI. However the CLI refuses to create the data-collection rule because the validation for the --data-flows option is too strict. Specifically, the validation for the "streams" property: https://github.com/Azure/azure-cli-extensions/blob/e2dad8c97f70c6b002749a7864f9dda2485498dd/src/monitor-control-service/azext_amcs/aaz/latest/monitor/data_collection/rule/_create.py#L157C9-L159C10

If I comment out line 158 (containing enum=...): nano ~/.azure/cliextensions/monitor-control-service/azext_amcs/aaz/latest/monitor/data_collection/rule/_create.py. Then my rule is created without issue.

Related command

az provider register --namespace "Microsoft.OperationalInsights"
az provider register --namespace "Microsoft.Insights"
az monitor log-analytics workspace create --resource-group "MyResourceGroup" --name "MyWorkspace"
az monitor log-analytics workspace table create --name "MyTable_CL" --resource-group "MyResourceGroup" --workspace-name "MyWorkspace" --columns TimeGenerated=datetime example=string
az monitor data-collection endpoint create --data-collection-endpoint-name "MyDce" --public-network-access "Enabled" --resource-group "MyResourceGroup"
az monitor data-collection rule create --data-collection-rule-name "MyDcr" --resource-group "MyResourceGroup" --data-collection-endpoint-id "/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/MyDce" --destinations "log-analytics[0]={name:'log-analytics-dest',workspace-resource-id:'/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.OperationalInsights/workspaces/MyWorkspace'}" --data-flows "[0]={streams:[Custom-MyTable_CL],outputStream:Custom-MyTable_CL,destinations:[log-analytics-dest],transformKql:source}" --stream-declarations "MyTable_CL.columns=[{name:TimeGenerated,type:datetime},{name:example,type:string}]"

Errors

unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'

Issue script & Debug output

az monitor data-collection rule create --data-collection-rule-name "MyDcr" --resource-group "MyResourceGroup" --data-collection-endpoint-id "/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/MyDce" --destinations "log-analytics[0]={name:'log-analytics-dest',workspace-resource-id:'/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.OperationalInsights/workspaces/MyWorkspace'}" --data-flows "[0]={streams:[Custom-MyTable_CL],outputStream:Custom-MyTable_CL,destinations:[log-analytics-dest],transformKql:source}" --stream-declarations "MyTable_CL.columns=[{name:TimeGenerated,type:datetime},{name:example,type:string}]" --debug

cli.knack.cli: Command arguments: ['monitor', 'data-collection', 'rule', 'create', '--data-collection-rule-name', 'MyDcr', '--resource-group', 'MyResourceGroup', '--data-collection-endpoint-id', '/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/MyDce', '--destinations', "log-analytics[0]={name:'log-analytics-dest',workspace-resource-id:'/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.OperationalInsights/workspaces/MyWorkspace'}", '--data-flows', '[0]={streams:[Custom-MyTable_CL],outputStream:Custom-MyTable_CL,destinations:[log-analytics-dest],transformKql:source}', '--stream-declarations', 'MyTable_CL.columns=[{name:TimeGenerated,type:datetime},{name:example,type:string}]', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x103d9d3a0>, <function OutputProducer.on_global_arguments at 0x103e48540>, <function CLIQuery.on_global_arguments at 0x103e7a020>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'monitor': ['azure.cli.command_modules.monitor', 'azext_amcs']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: monitor                   0.530        22        65
cli.azure.cli.core: Total (1)                 0.530        22        65
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: monitor-control-service      0.018        10        30  /Users/foo/.azure/cliextensions/monitor-control-service
cli.azure.cli.core: Total (1)                 0.018        10        30
cli.azure.cli.core: Loaded 31 groups, 95 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : monitor data-collection rule create
cli.azure.cli.core: Command table: monitor data-collection rule create
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x104bb6ca0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/Users/foo/.azure/commands/2024-03-19.17-10-28.monitor_data-collection_rule_create.64328.log'.
az_command_data_logger: command args: monitor data-collection rule create --data-collection-rule-name {} --resource-group {} --data-collection-endpoint-id {} --destinations {} --data-flows {} --stream-declarations {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x104c0b9c0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x104c0ba60>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x104c0bba0>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 589, in execute
    parsed_args = self.parser.parse_args(args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/knack/parser.py", line 261, in parse_args
    return super().parse_args(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1869, in parse_args
    args, argv = self.parse_known_args(args, namespace)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1902, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2096, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2073, in consume_positionals
    take_action(action, args)
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1978, in take_action
    action(self, namespace, argument_values, option_string)
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1241, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1902, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2096, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2073, in consume_positionals
    take_action(action, args)
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1978, in take_action
    action(self, namespace, argument_values, option_string)
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1241, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1902, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2096, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2073, in consume_positionals
    take_action(action, args)
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1978, in take_action
    action(self, namespace, argument_values, option_string)
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1241, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1902, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2096, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2073, in consume_positionals
    take_action(action, args)
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1978, in take_action
    action(self, namespace, argument_values, option_string)
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1241, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1902, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2114, in _parse_known_args
    start_index = consume_optional(start_index)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2054, in consume_optional
    take_action(action, args, option_string)
  File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1978, in take_action
    action(self, namespace, argument_values, option_string)
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 394, in __call__
    self.setup_operations(dest_ops, values)
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 435, in setup_operations
    raise ex
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 420, in setup_operations
    data = action.format_data(value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 291, in format_data
    result[key] = action.format_data(value)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 478, in format_data
    result.append(action.format_data(value))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 156, in format_data
    return cls._schema.enum[data]
           ~~~~~~~~~~~~~~~~^^^^^^
  File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg.py", line 74, in __getitem__
    raise azclierror.InvalidArgumentValueError(
azure.cli.core.azclierror.InvalidArgumentValueError: unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'

cli.azure.cli.core.azclierror: unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'
az_command_data_logger: unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x104bb6f20>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 0.749 seconds (init: 0.147, invoke: 0.602)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3863 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/bin/python /opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /Users/foo/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

The rule should be created without error

Environment Summary

azure-cli                         2.58.0

core                              2.58.0
telemetry                          1.1.0

Extensions:
monitor-control-service               1.0.1

Dependencies:
msal                              1.26.0
azure-mgmt-resource             23.1.0b2

Python location '/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/bin/python'
Extensions directory '/Users/foo/.azure/cliextensions'

Python (Darwin) 3.11.8 (main, Feb  6 2024, 21:21:21) [Clang 15.0.0 (clang-1500.1.0.2.5)]

Legal docs and information: aka.ms/AzureCliLegal


Your CLI is up-to-date.

Additional context

No response

Joris-van-der-Wel avatar Mar 19 '24 16:03 Joris-van-der-Wel

Thank you for opening this issue, we will look into it.

yonzhan avatar Mar 19 '24 16:03 yonzhan

@Joris-van-der-Wel thanks Joris for this difference. Not sure how azure portal deal with this but azure cli generates code based on service api doc, which can be referred here: https://github.com/Azure/azure-rest-api-specs/blob/42869ad83518d47ab2a24c07d888cb29fdeb43fd/specification/monitor/resource-manager/Microsoft.Insights/stable/2022-06-01/dataCollectionRules_API.json#L1135-L1148 .

If that's the common user case, we'll let service team know the gap and whether they need to remove this enum validations.

AllyW avatar Mar 28 '24 01:03 AllyW

I think that jsonschema definition is indeed wrong. I think the proper schema would be something like:

{
    "streams": {
        "description": "List of streams for this data flow.",
        "type": "array",
        "items": {
            "pattern": "^(Microsoft-Event|Microsoft-InsightsMetrics|Microsoft-Perf|Microsoft-Syslog|Microsoft-WindowsEvent|Custom-[\\w-]+)$",
            "type": "string"
        }
    }
}

Also, it seems to me that the actual API server uses a different schema to validate with? Would it not be possible to copy that schema? For example if I pass a wrong value to the "streams" property, the server will respond with:

{
    "error": {
        "code": "InvalidPayload",
        "message": "Data collection rule is invalid",
        "details": [
            {
                "code": "InvalidStream",
                "message": "'Streams' stream 'Foo' must be a custom stream or one of the allowed streams.",
                "target": "Properties.DataFlows[0].Streams[0]"
            }
        ]
    }
}

Similarly, if I use an invalid property name for the stream declaration, I get this error:

{
    "error": {
        "code": "InvalidPayload",
        "message": "Data collection rule is invalid",
        "details": [
            {
                "code": "InvalidStreamDeclaration",
                "message": "'' stream 'Foo' must be a custom stream.",
                "target": "Properties.StreamDeclarations[0].Key"
            },
            {
                "code": "InvalidStreamDeclaration",
                "message": "'' is not a valid custom stream name. It should start with 'Custom-' followed by a letter and then any combination of alphanumeric characters and '-', '_'",
                "target": "Properties.StreamDeclarations[0].Key"
            }
        ]
    }
}

Custom streams are supported by Azure in all of the other ways that I have tried:

Azure Portal

  1. Create Log Analytics workspace
  2. "Tables" => "Create" => "New custom log (DCR-based)"
  3. Enter a name
  4. "Create a new data collection rule" => Pick a name => "Done"
  5. Select an "data collection endpoint"
  6. "Next"
  7. Upload an example json: [{"TimeGenerated":"2024-03-28T11:00:00Z","hello":"world"}]
  8. "Next"
  9. Open the DevTools (F12) => Network => XHR
  10. "Create"
  11. In the dev tools right click on the PUT request to example-dcr?api-version=2021-09-01-preview&ignoreMissingTables=true and copy the PUT-data

This will look something like this:

{
    "id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionRules/example-dcr",
    "location": "westeurope",
    "type": "Microsoft.Insights/dataCollectionRules",
    "name": "example-dcr",
    "properties": {
        "dataSources": {},
        "destinations": {
            "logAnalytics": [{
                "workspaceResourceId": "/subscriptions/XXX/resourcegroups/joris/providers/microsoft.operationalinsights/workspaces/XXX",
                "workspaceId": "XXX",
                "name": "123456789abcdef123456789abcdef12"
            }]
        },
        "dataFlows": [{
            "streams": ["Custom-Example_CL"],
            "transformKql": "source",
            "destinations": ["123456789abcdef123456789abcdef12"],
            "outputStream": "Custom-Example_CL"
        }],
        "streamDeclarations": {
            "Custom-Example_CL": {
                "columns": [
                    {
                        "name": "TimeGenerated",
                        "type": "datetime"
                    },
                    {
                        "name": "hello",
                        "type": "string"
                    }
                ]
            }
        },
        "dataCollectionEndpointId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionEndpoints/XXX"
    }
}

Azure CLI (list)

Next, try to use: az monitor data-collection rule list. This will demonstrate the same format for "dataFlows":

[
    {
        "dataCollectionEndpointId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionEndpoints/XXX",
        "dataFlows": [{
            "destinations": ["123456789abcdef123456789abcdef12"],
            "outputStream": "Custom-Example_CL",
            "streams": ["Custom-Example_CL"],
            "transformKql": "source"
        }],
        "dataSources": {},
        "destinations": {
            "logAnalytics": [{
                "name": "123456789abcdef123456789abcdef12",
                "workspaceId": "XXX",
                "workspaceResourceId": "/subscriptions/XXX/resourcegroups/XXX/providers/microsoft.operationalinsights/workspaces/XXX"
            }]
        },
        "etag": "\"XXX\"",
        "id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionRules/example-dcr",
        "immutableId": "dcr-3456789abcdef123456789abcdef1245",
        "location": "westeurope",
        "name": "example-dcr",
        "provisioningState": "Succeeded",
        "resourceGroup": "XXX",
        "streamDeclarations": {
            "Custom-Example_CL": {
                "columns": [
                    {
                        "name": "TimeGenerated",
                        "type": "datetime"
                    },
                    {
                        "name": "hello",
                        "type": "string"
                    }
                ]
            }
        },
        "systemData": {
            "createdAt": "2024-03-28T11:00:00Z",
            "createdBy": "jvanderwel@XXX",
            "createdByType": "User",
            "lastModifiedAt": "2024-03-28T11:00:00Z",
            "lastModifiedBy": "jvanderwel@XXX",
            "lastModifiedByType": "User"
        },
        "type": "Microsoft.Insights/dataCollectionRules"
    }
]

Azure Resource Manager

I can deploy a bicep template that uses a custom stream without issue:

resource workspace 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
    name: 'myworkspace'
    location: 'westeurope'
}

resource dce 'Microsoft.Insights/dataCollectionEndpoints@2022-06-01' = {
    name: 'mydce'
    location: 'westeurope'
}

resource table 'Microsoft.OperationalInsights/workspaces/tables@2022-10-01' = {
    parent: workspace
    name: 'Example_CL'

    properties: {
        schema: {
            name: 'Example_CL'
            columns: [
                {name: 'TimeGenerated', type: 'datetime'}
                {name: 'hello', type: 'string'}
            ]
        }
    }
}

resource dcr 'Microsoft.Insights/dataCollectionRules@2022-06-01' = {
    name: 'mydcr'
    location: 'westeurope'
    properties: {
        dataCollectionEndpointId: dce.id
        dataFlows: [
            {
                destinations: ['log-analytics-dest']
                outputStream: 'Custom-Example_CL'
                streams: ['Custom-Example_CL']
                transformKql: 'source'
            }
        ]
        destinations: {
            logAnalytics: [
                {
                    name: 'log-analytics-dest'
                    workspaceResourceId: workspace.id
                }
            ]
        }
        streamDeclarations: {
            'Custom-Example_CL': {
                columns: [
                    {name: 'TimeGenerated', type: 'datetime'}
                    {name: 'hello', type: 'string'}
                ]
            }
        }
    }
}

Joris-van-der-Wel avatar Mar 28 '24 11:03 Joris-van-der-Wel

@Joris-van-der-Wel we will add support for enumeration arguments which skips validation recently. Please follow the linked pr for future development and releases.

AllyW avatar May 31 '24 11:05 AllyW