[FR] CI Job to Sync ES|QL Custom Fields with Prebuilt Filterlist for Telemetry

[terrancedejesus]

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

At the moment, when using ES|QL for writing detection rule queries, often we use aggregate functions, eval or pre-processing functions (grok and dissect) to create useful fields for our filters.

In this instance, those fields are not available in global alert telemetry, which relies on a static filterlist for determining what fields to ingest from the alerts.

Desired Solution

As such, we must develop a CI job that loads the ES|QL rules, reviews custom fields and adds those to the filterlist. The CI job would run on merges into main only.

Considered Alternatives

No alternatives considered. This suggestion is post conversation with Security Data Analytics team.

Additional Context

Related to https://github.com/elastic/ia-trade-team/issues/101

[Mikaayenson]

We will need to consider how to handle the internal list in CI. We may want to kickoff a job that runs in CI in a different repo.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic[bot]]

This has been closed due to inactivity. If you feel this is an error, please re-open and include a justifying comment.