elastic
[Bug] Missing Related Integrations and Required Fields for ESQL Rules
Describe the Bug
Since we do not have an ESQL parser that can pull out event.datasource and we do not include index since it is duplicative with the query, we are not able to populated the related_integration build time field.
This means that in the security solution, ESQL rules are missing this information.
One idea not fully explored would be to add the related integration information to the toml file (as a stop gap until we have an esql parser), but this would have to be fully explored with the team for viability.
We could potentially add the metadata to the toml.
[[rule.related_integrations]]
package="azure"
integration="signinlogs"
version = "N/A"
Note: This would increase toil for our detection engineers and require us to maintain.
To Reproduce
- Select any ESQL rule
- run the
view-rulecommand with the rule - See the
related_integrationsfield is missing in the output.
Expected Behavior
Related Integrations should be available in the security solution.
Screenshots
Desktop - OS
None
Desktop - Version
No response
Additional Context
One approach may be to manually add the related_integrations field and skip the required version field so that it can be programmatically picked up.
diff --git a/detection_rules/rule.py b/detection_rules/rule.py
index 7dd295feb..ee4ab00d3 100644
--- a/detection_rules/rule.py
+++ b/detection_rules/rule.py
@@ -1231,7 +1231,9 @@ class TOMLRuleContents(BaseRuleContents, MarshmallowDataclassMixin):
field_name = "related_integrations"
package_integrations = obj.get(field_name, [])
- if not package_integrations and self.metadata.integration:
+ # missing if any versions are N/A
+ missing_package_versions = any(pk['version'] == "N/A" for pk in package_integrations)
+ if (not package_integrations or (package_integrations and missing_package_versions)) and self.metadata.integration:
packages_manifest = load_integrations_manifests()
current_stack_version = load_current_package_version()
@@ -1239,8 +1241,10 @@ class TOMLRuleContents(BaseRuleContents, MarshmallowDataclassMixin):
if (isinstance(self.data, QueryRuleData) or isinstance(self.data, MachineLearningRuleData)):
if (self.data.get('language') is not None and self.data.get('language') != 'lucene') or \
self.data.get('type') == 'machine_learning':
- package_integrations = self.get_packaged_integrations(self.data, self.metadata,
- packages_manifest)
+
+ if not package_integrations:
+ package_integrations = self.get_packaged_integrations(self.data, self.metadata,
+ packages_manifest)
if not package_integrations:
return
@@ -1259,6 +1263,7 @@ class TOMLRuleContents(BaseRuleContents, MarshmallowDataclassMixin):
policy_templates = version_data.get("policy_templates", [])
if package["integration"] not in policy_templates:
+ # TODO: Error when integration is not a policy template because we just didnt update the manifest
del package["integration"]
# remove duplicate entries
@@ -1266,6 +1271,7 @@ class TOMLRuleContents(BaseRuleContents, MarshmallowDataclassMixin):
d for d in package_integrations}.values())
obj.setdefault("related_integrations", package_integrations)
+
def _convert_add_required_fields(self, obj: dict) -> None:
"""Add restricted field required_fields to the obj, derived from the query AST."""
if isinstance(self.data, QueryRuleData) and self.data.language != 'lucene':
diff --git a/detection_rules/rule_validators.py b/detection_rules/rule_validators.py
index 5fe957ec1..d94e529ea 100644
--- a/detection_rules/rule_validators.py
+++ b/detection_rules/rule_validators.py
@@ -597,11 +597,18 @@ class ESQLValidator(QueryValidator):
def validate(self, data: 'QueryRuleData', meta: RuleMeta) -> None:
"""Validate an ESQL query while checking TOMLRule."""
# temporarily override to NOP until ES|QL query parsing is supported
+ self.validate_integration(data, meta, data.related_integrations)
def validate_integration(self, data: QueryRuleData, meta: RuleMeta, package_integrations: List[dict]) -> Union[
ValidationError, None, ValueError]:
- # return self.validate(data, meta)
- pass
+ # Force all esql rules to manually specify the related_integrations
+ if not package_integrations:
+ raise ValueError("ES|QL rules must specify related_integrations")
+
+ for package in package_integrations:
+ version = package.version.strip("^")
+ if version != "N/A" and not Version.is_valid(version):
+ raise ValueError("ES|QL rules must specify a version for related_integrations or N/A to get the latest")
def extract_error_field(source: str, exc: Union[eql.EqlParseError, kql.KqlParseError]) -> Optional[str]:
diff --git a/rules/integrations/azure/credential_access_entra_signin_brute_force_microsoft_365.toml b/rules/integrations/azure/credential_access_entra_signin_brute_force_microsoft_365.toml
index 7e451f3a7..b41e05538 100644
--- a/rules/integrations/azure/credential_access_entra_signin_brute_force_microsoft_365.toml
+++ b/rules/integrations/azure/credential_access_entra_signin_brute_force_microsoft_365.toml
@@ -6,6 +6,7 @@ min_stack_comments = "ES|QL not available until 8.13.0 in technical preview."
min_stack_version = "8.13.0"
updated_date = "2025/01/15"
+
[rule]
author = ["Elastic"]
description = """
@@ -108,6 +109,10 @@ from logs-azure.signinlogs*
| where (login_source_count >= 20 or failed_login_count >= 20)
'''
+[[rule.related_integrations]]
+package="azure"
+integration="signinlogs"
+version = "N/A"
[[rule.threat]]
framework = "MITRE ATT&CK"
diff --git a/rules/windows/collection_email_powershell_exchange_mailbox.toml b/rules/windows/collection_email_powershell_exchange_mailbox.toml
index 1d39e1988..af74334fe 100644
--- a/rules/windows/collection_email_powershell_exchange_mailbox.toml
+++ b/rules/windows/collection_email_powershell_exchange_mailbox.toml
@@ -95,6 +95,7 @@ tags = [
"Data Source: SentinelOne",
"Data Source: Microsoft Defender for Endpoint",
"Data Source: System",
+
]
timestamp_override = "event.ingested"
type = "eql"
[!CAUTION] With the new docs system, they will be picking up the toml and auto converting to markdown. Since this approach uses a buildtime automation, this will not work for docs.
We might be able to leverage Kibana's Lexer/parser (ref: https://github.com/elastic/kibana/pull/213006/files) 🤔
Here is the short outcome after discussing with the team:
- This workaround will only address related integrations. We will still have issues pulling required_fields since we do not have a parser.
- It creates toil for our authors to maintain the actual fields which are otherwise built automatically for all other rule types
- This may inadvertently set precedence to continue adding workarounds instead of prioritizing a parser.
At the moment we do not have plans to implement this, and will instead hold until https://github.com/elastic/security-team/issues/11554 is provided.
23 Oct 2025
With ESQL Remote Validation merged https://github.com/elastic/detection-rules/pull/4955, this can now be used to populate related_integrations and required_fields for ESQL rules.
Release Fleet workflow would have the below modifications from esql_validation to allow build packages for ESQL rules to populate related_integrations and required_fields for ESQL rules.
- Checkout elastic-container keep the branching logic from the esql-validation intact
- Build and run containers
- Get API Key and setup auth
- And add these variables DR_REMOTE_ESQL_VALIDATION: "true" DR_CLOUD_ID: ${{ secrets.dr_cloud_id }} DR_KIBANA_URL: ${{ secrets.dr_cloud_id }} DR_ELASTICSEARCH_URL: ${{ secrets.dr_cloud_id }} DR_API_KEY: ${{ secrets.dr_api_key }} DR_IGNORE_SSL_ERRORS: ${{ secrets.dr_cloud_id }}
- So Build release package will pickup remote ESQL validation.
