ConvertSentinelRuleFrom-Yaml.ps1 converts to a JSON format that does not conform to ARM format

[ingest0x]

Describe the bug When using the ConvertSentinelRuleFrom-Yaml.ps1 script to convert a folder of YAML analytics rules to JSON, the CI/CD process fails due to an issue with the formatting (see error message below).

When exporting the same rules from Sentinel via GUI and coparing the formating of the exported JSON to the converted JSON the files are very different. I'm wondering if this is causing the problem? I can deploy the exported JSON back into Sentinel using VSCode + Azure DevOps + Repository in Sentinel absolutely fine.

To Reproduce Steps to reproduce the behavior:

1. Use ConvertSentinelRuleFrom-Yaml.ps1 to convert a folder of YAMLs into JSONs
2. Attempt to push to Azure Devops repository connected to Azure Sentinel workspace
3. Instead receive error:

2022-11-14T16:18:21.9220516Z [Warning] Failed to deploy D:\a\1\s\Detections\NRT_AWS_ConsoleLogonWithoutMFA.json with error: 4:18:21 PM - Error: Code=InvalidTemplate; Message=Deployment template validation failed: 'The template resource '<redacted-workspace-name>/Microsoft.SecurityInsights/' for type 'Microsoft.OperationalInsights/workspaces/providers/alertRules' at line '15' and column '81' has incorrect segment lengths. A nested resource type must have identical number of segments as its resource name. A root resource type must have segment length one greater than its resource name. Please see https://aka.ms/arm-template/#resources for usage details.'.

4. If you export a rule from Sentinel and compare it to the "converted" rule the formatting & fields are very different.

Expected behavior Expect to see analytics rules appear in Sentinel

Desktop (please complete the following information): Windows 10 22H2

Additional context Add any other context about the problem here.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[v-rbajaj]

Hi @ingest0x, We wanted to check on the status of Issue https://github.com/Azure/Azure-Sentinel/issues/6621. The issue is pending for more than expected days. Please let us know if you need any assistance to review this Issue. As per our standard operating procedures if no response is received in the next 7 business days, we will close this Issue. Thank you for your cooperation.

[ingest0x]

I mean, all the comments have been automated "thank you" comments - nothing has actually been done about the isssue raised... So yea, would be nice to get this resolved for the community.

[v-vdixit]

Hi @ingest0x we are just checking on latest active issues thanks for confirming we will work on this with the team and resolve this, thanks!

[v-vdixit]

Hi @ingest0x we have reached out to the concerned team for this issue, will update you once we hear back from them, thanks!

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[v-vdixit]

Hi @ingest0x we are waiting for update from the concerned team, will update you once we hear back from them, thanks!

[v-vdixit]

Hi @ingest0x we are still waiting to hear back from concerned team, thanks!

[v-vdixit]

We are still waiting for update from the concerned team, thanks

[v-vdixit]

We are waiting for update from script author, will update you once we hear back from them, thanks!

[v-vdixit]

Hi @ingest0x We are still waiting for update from the script author, thanks

[v-vdixit]

Hi @ingest0x we are continuously following up with the author, we will update you once we hear back from the author, thanks!

[v-rbajaj]

Hi @ingest0x, we are still waiting for the response from author on this issue, we will get back to you once we have update.

[v-rbajaj]

Hi @ingest0x, waiting for update from the internal team.

[v-rbajaj]

Hi @ingest0x, waiting for update from the internal team.

[v-rbajaj]

Hi @ingest0x, waiting for update from the internal team.

[v-rbajaj]

We are trying to reproduce this issue in our end.

[ingest0x]

OK, thanks

[v-rbajaj]

Hi @ingest0x, We are trying to reproduce this issue.The powershell script converts the yaml into JSON but when we deploy the JSONs, the validations fails, though we have fixed few errors but still not fine enough.

However we aren't able to produce the error which is mentioned in this issue. Can you please try from your end and see if the you are getting the same error while deployment (the analytical rule might have got updated after this issue is raised).

[azurekid]

Hi, @ingest0x and @v-rbajaj,

As far as I know this script directly converts the values of the yaml into the JSON template. The issue with this is that the time format for the durations in the yaml 1h is different than the supported ISO 8601 format in the JSON file PT1H.

Also the conversion from the yaml value (gt, lt, eq and ne) is missing as the JSON file only supports the values (GreatherThan, LessThan, Equal and NotEqual.)

I have created my own script that I use in a GitHub Action to convert the yml files into deployable ARM templates. Maybe you can use this as a reference.

https://github.com/SecureHats/YamlTo-Arm

The example can be found in this file: https://github.com/SecureHats/YamlTo-Arm/blob/main/modules/HelperFunctions.psm1 on line 114 and 124

I was able to reproduce the issue, and also tried it with my own GitHub action which seams to output a correctly working file.

Hopefully this information helps.

Rogier Dijkman (MVP Security)

[v-rbajaj]

Hi, @ingest0x and @v-rbajaj,

As far as I know this script directly converts the values of the yaml into the JSON template. The issue with this is that the time format for the durations in the yaml 1h is different than the supported ISO 8601 format in the JSON file PT1H.

Also the conversion from the yaml value (gt, lt, eq and ne) is missing as the JSON file only supports the values (GreatherThan, LessThan, Equal and NotEqual.)

I have created my own script that I use in a GitHub Action to convert the yml files into deployable ARM templates. Maybe you can use this as a reference.

https://github.com/SecureHats/YamlTo-Arm

The example can be found in this file: https://github.com/SecureHats/YamlTo-ArmArm/blob/main/modules/HelperFunctions.psm1 on line 114 and 124

I was able to reproduce the issue, and also tried it with my own GitHub action which seams to output a correctly working file.

Hopefully this information helps.

Rogier Dijkman (MVP Security)

Thanks Rogier for providing the solution. If this works for @ingest0x then we can update the original script.

[ingest0x]

Thanks @azurekid - I actually "fixed" this myself some time in December - I say "fixed" because I'm not proficient in Powershell so made the changes I needed to get the script working and has been for some time now. I raised this issue here to have the issue solved by a competent PS developer for the betterment of others.

I will review your script and see how I can improve mine, if that's OK :)