azure-cli-extensions
azure-cli-extensions copied to clipboard
Published
20 hours ago •
Azure
Azure
Can not create data-collection rule with a custom stream
Describe the bug
I am attempting to create a data collection rule in azure monitor, that is intended to be used with a custom application that calls the Logs Ingestion API. I have created a logs-analytics workspace containing a custom table. My data collection rule should be configured without any data sources, instead a custom stream for my table is declared.
This all works fine when I configure this via the Azure Portal GUI. However the CLI refuses to create the data-collection rule because the validation for the --data-flows option is too strict. Specifically, the validation for the "streams" property: https://github.com/Azure/azure-cli-extensions/blob/e2dad8c97f70c6b002749a7864f9dda2485498dd/src/monitor-control-service/azext_amcs/aaz/latest/monitor/data_collection/rule/_create.py#L157C9-L159C10
If I comment out line 158 (containing enum=...):
nano ~/.azure/cliextensions/monitor-control-service/azext_amcs/aaz/latest/monitor/data_collection/rule/_create.py.
Then my rule is created without issue.
Related command
az provider register --namespace "Microsoft.OperationalInsights"
az provider register --namespace "Microsoft.Insights"
az monitor log-analytics workspace create --resource-group "MyResourceGroup" --name "MyWorkspace"
az monitor log-analytics workspace table create --name "MyTable_CL" --resource-group "MyResourceGroup" --workspace-name "MyWorkspace" --columns TimeGenerated=datetime example=string
az monitor data-collection endpoint create --data-collection-endpoint-name "MyDce" --public-network-access "Enabled" --resource-group "MyResourceGroup"
az monitor data-collection rule create --data-collection-rule-name "MyDcr" --resource-group "MyResourceGroup" --data-collection-endpoint-id "/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/MyDce" --destinations "log-analytics[0]={name:'log-analytics-dest',workspace-resource-id:'/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.OperationalInsights/workspaces/MyWorkspace'}" --data-flows "[0]={streams:[Custom-MyTable_CL],outputStream:Custom-MyTable_CL,destinations:[log-analytics-dest],transformKql:source}" --stream-declarations "MyTable_CL.columns=[{name:TimeGenerated,type:datetime},{name:example,type:string}]"
Errors
unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'
Issue script & Debug output
az monitor data-collection rule create --data-collection-rule-name "MyDcr" --resource-group "MyResourceGroup" --data-collection-endpoint-id "/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/MyDce" --destinations "log-analytics[0]={name:'log-analytics-dest',workspace-resource-id:'/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.OperationalInsights/workspaces/MyWorkspace'}" --data-flows "[0]={streams:[Custom-MyTable_CL],outputStream:Custom-MyTable_CL,destinations:[log-analytics-dest],transformKql:source}" --stream-declarations "MyTable_CL.columns=[{name:TimeGenerated,type:datetime},{name:example,type:string}]" --debug
cli.knack.cli: Command arguments: ['monitor', 'data-collection', 'rule', 'create', '--data-collection-rule-name', 'MyDcr', '--resource-group', 'MyResourceGroup', '--data-collection-endpoint-id', '/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/MyDce', '--destinations', "log-analytics[0]={name:'log-analytics-dest',workspace-resource-id:'/subscriptions/UUID/resourceGroups/MyResourceGroup/providers/Microsoft.OperationalInsights/workspaces/MyWorkspace'}", '--data-flows', '[0]={streams:[Custom-MyTable_CL],outputStream:Custom-MyTable_CL,destinations:[log-analytics-dest],transformKql:source}', '--stream-declarations', 'MyTable_CL.columns=[{name:TimeGenerated,type:datetime},{name:example,type:string}]', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x103d9d3a0>, <function OutputProducer.on_global_arguments at 0x103e48540>, <function CLIQuery.on_global_arguments at 0x103e7a020>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'monitor': ['azure.cli.command_modules.monitor', 'azext_amcs']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: monitor 0.530 22 65
cli.azure.cli.core: Total (1) 0.530 22 65
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: monitor-control-service 0.018 10 30 /Users/foo/.azure/cliextensions/monitor-control-service
cli.azure.cli.core: Total (1) 0.018 10 30
cli.azure.cli.core: Loaded 31 groups, 95 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : monitor data-collection rule create
cli.azure.cli.core: Command table: monitor data-collection rule create
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x104bb6ca0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/Users/foo/.azure/commands/2024-03-19.17-10-28.monitor_data-collection_rule_create.64328.log'.
az_command_data_logger: command args: monitor data-collection rule create --data-collection-rule-name {} --resource-group {} --data-collection-endpoint-id {} --destinations {} --data-flows {} --stream-declarations {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x104c0b9c0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x104c0ba60>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x104c0bba0>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 589, in execute
parsed_args = self.parser.parse_args(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/knack/parser.py", line 261, in parse_args
return super().parse_args(args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1869, in parse_args
args, argv = self.parse_known_args(args, namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1902, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2096, in _parse_known_args
positionals_end_index = consume_positionals(start_index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2073, in consume_positionals
take_action(action, args)
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1978, in take_action
action(self, namespace, argument_values, option_string)
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1241, in __call__
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1902, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2096, in _parse_known_args
positionals_end_index = consume_positionals(start_index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2073, in consume_positionals
take_action(action, args)
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1978, in take_action
action(self, namespace, argument_values, option_string)
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1241, in __call__
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1902, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2096, in _parse_known_args
positionals_end_index = consume_positionals(start_index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2073, in consume_positionals
take_action(action, args)
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1978, in take_action
action(self, namespace, argument_values, option_string)
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1241, in __call__
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1902, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2096, in _parse_known_args
positionals_end_index = consume_positionals(start_index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2073, in consume_positionals
take_action(action, args)
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1978, in take_action
action(self, namespace, argument_values, option_string)
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1241, in __call__
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1902, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2114, in _parse_known_args
start_index = consume_optional(start_index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 2054, in consume_optional
take_action(action, args, option_string)
File "/opt/homebrew/Cellar/[email protected]/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/argparse.py", line 1978, in take_action
action(self, namespace, argument_values, option_string)
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 394, in __call__
self.setup_operations(dest_ops, values)
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 435, in setup_operations
raise ex
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 420, in setup_operations
data = action.format_data(value)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 291, in format_data
result[key] = action.format_data(value)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 478, in format_data
result.append(action.format_data(value))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg_action.py", line 156, in format_data
return cls._schema.enum[data]
~~~~~~~~~~~~~~~~^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/core/aaz/_arg.py", line 74, in __getitem__
raise azclierror.InvalidArgumentValueError(
azure.cli.core.azclierror.InvalidArgumentValueError: unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'
cli.azure.cli.core.azclierror: unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'
az_command_data_logger: unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x104bb6f20>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 0.749 seconds (init: 0.147, invoke: 0.602)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3863 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/bin/python /opt/homebrew/Cellar/azure-cli/2.58.0/libexec/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /Users/foo/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.
Expected behavior
The rule should be created without error
Environment Summary
azure-cli 2.58.0
core 2.58.0
telemetry 1.1.0
Extensions:
monitor-control-service 1.0.1
Dependencies:
msal 1.26.0
azure-mgmt-resource 23.1.0b2
Python location '/opt/homebrew/Cellar/azure-cli/2.58.0/libexec/bin/python'
Extensions directory '/Users/foo/.azure/cliextensions'
Python (Darwin) 3.11.8 (main, Feb 6 2024, 21:21:21) [Clang 15.0.0 (clang-1500.1.0.2.5)]
Legal docs and information: aka.ms/AzureCliLegal
Your CLI is up-to-date.
Additional context
No response
