elastic
[Bug] Using the CLI to export esql (ES|QL) rules from Kibana results in ValidationError if using metadata according to documentation
Describe the Bug
The CLI python -m detection_rules kibana export-rules doesnt work with a simple esql rule, where metadata is set according to official documentation. It always leads to:
marshmallow.exceptions.ValidationError: {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'_schema': ["Rule: test_fberg_esql contains a non-aggregate query without metadata fields '_id', '_version', and '_index' -> Add 'metadata _id, _version, _index' to the from command or add an aggregate function."]}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}
To Reproduce
- Create a simple esql rule:
- Try to export it with the CLI and
kibana export-rules - Leads to ValidationError
Expected Behavior
No ValidationError since esql metadata is set according to documentation.
Edit: I fixed this via PR where we validate the order and allow any order of metadata
Screenshots
No response
Desktop - OS
None
Desktop - Version
No response
Additional Context
No response
Just adding for context for reviewers on the ^ PR. This issue can be more easily tested via DaC commands (loading just a specific rule to the rule loader), but the fundamental issue is with the ES|QL validation for rules passing schema validation rather than any DaC command.
Another testing example illustrating issue:
Closing as this issue has been resolved via https://github.com/elastic/detection-rules/pull/4956