[FR] Deprecate Experimental ML Logic

[Mikaayenson]

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

We still have logic marked as experimental within our repo that should be deprecated and ultimately removed. This logic was intended to support ml functionality that is no longer supported.

(detection-rules-build) ➜  detection-rules git:(main) ✗ python -m detection_rules es  experimental ml -h
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Elasticsearch client:
Options:
  --ignore-ssl-errors TEXT
  -et, --timeout INTEGER    Timeout for elasticsearch client
  -ep, --es-password TEXT
  -eu, --es-user TEXT
  --elasticsearch-url TEXT
  --api-key TEXT
  --cloud-id TEXT


* experimental commands are use at your own risk and may change without warning *

Usage: detection_rules es experimental ml [OPTIONS] COMMAND [ARGS]...

  Experimental machine learning commands.

Options:
  -h, --help  Show this message and exit.

Commands:
  check-files               Check ML model files on an elasticsearch...
  delete-job                Remove experimental ML jobs.
  remove-model              Remove ML model files.
  remove-scripts-pipelines  Remove ML scripts and pipeline files.
  setup                     Upload ML model and dependencies to enrich data.
  upload-job                Upload experimental ML jobs.

When you try to use this existing code today, it throws an error because it uses a deprecated way to call the ES API.

detection_rules es experimental ml upload-job my-job.json

TypeError: Positional arguments can't be used with Elasticsearch API methods. Instead only use keyword arguments.

Desired Solution

These CLI commands should be officially removed. We should tag the commit for folks to have a reference point in case they still want to use this logic.

Considered Alternatives

We could update the call to supply the kwargs func(job_id=name, body=body) here as a starting point, but with the additional context below, this capability should no longer be supported.

Additional Context

We now have ML jobs supported via integrations.

Additionally, the original issue where the experimental ML features were merged in states that the ML release processes are to be managed via a different internal repo (dremel), which also has been deprecated with the statement:

NOTE: We are no longer using dreml to ship experimental machine learning models. All existing models have been/are being delivered via integration packages or other mechanisms. Please get in touch with the Security ML team for more information on packaging ML models.

[shashank-elastic]

We have just provide deprecation warnings and will remove the commands on May 1 2025. Reopening this issue to track the removal.

Moving it to Q4 2024

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic[bot]]

This has been closed due to inactivity. If you feel this is an error, please re-open and include a justifying comment.

[shashank-elastic]

We are keeping this open actually deprecate the code on May 1

[shashank-elastic]

Completed via PR https://github.com/elastic/detection-rules/pull/4669