kibana [R&D] Stateful/stateless test harness options for observability

Currently to run tests in both a stateful and serverless context, the current recommendation is to "copy the tests that should run in stateful and serverless", which requires teams to maintain those files separately, as well. We would like the ability to tell CI to run a selection of tests in both contexts, rather than needing to copy/paste the files.

This issue is a time-boxed R&D for someone to look into whether we can create some kind of simple "test harness" wrapper for running a selection of our functional tests in both contexts, that would work for our observability purposes.

Existing Tests

Here's the current code coverage for the SLO and o11y alerting features for both serverless and stateful environments:

E2E Functional tests
- SLO embeddable tests
  - Stateful: x-pack/test/functional/apps/slo/embeddables
- Alerting
  - Serverless
    - Rules page: x-pack/test/observability_functional/apps/observability/pages/rules_page.ts
    - Rule details page: x-pack/test/observability_functional/apps/observability/pages/rule_details_page.ts
    - Alerts page: x-pack/test/observability_functional/apps/observability/pages/alerts
    - Overview page alerts table: x-pack/test/observability_functional/apps/observability/pages/overview/alert_table.ts
API Integration tests
- Alerting:
  - Serverless
    - Custom threshold rule tests: x-pack/test_serverless/api_integration/test_suites/observability/custom_threshold_rule
    - SLO burn rate rules: x-pack/test_serverless/api_integration/test_suites/observability/burn_rate_rule
    - ES Query rule: x-pack/test_serverless/api_integration/test_suites/observability/es_query_rule
  - Stateful
    - Custom threshold rule tests: x-pack/test/alerting_api_integration/observability/custom_threshold_rule
    - Metric threshold rule tests: x-pack/test/alerting_api_integration/observability/metric_threshold_rule.ts
- SLO API tests
  - Serverless: x-pack/test_serverless/api_integration/test_suites/observability/slos
  - Stateful: x-pack/test/api_integration/apis/slos

Todo

[x] Get feedback from AppEx-qa and other solution teams
- [x] AppEx-qa: @pheyos
- [x] Security team: @MadameSheema @banderror
[x] Make a list of current functional and api integration tests for SLO and o11y alerting features
[x] Make a POC of reusing API integration (FTR) tests from x-pack/test_serverless/api_integration/test_suites/observability/burn_rate_rule/burn_rate_rule.ts
- [x] https://github.com/elastic/kibana/pull/182556
- [x] https://github.com/elastic/kibana/pull/183113
[ ] Make a POC of deployment agnostic functional tests
- [ ] Reuse Security's solution using Cypress
- [ ] Investigate how can reuse FTR tests
[ ] Make a plan to migrate existing tests to the new location
- [ ] Integration tests
- [ ] Functional tests

Mar 27 '24 14:03 jasonrhodes

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

Mar 27 '24 14:03 elasticmachine

The approach we could take is to create an abstraction that basically injects dependencies for each environment (Stateless vs Stateful), then we would refactor our tests to use the abstractions instead of writing tests in both places.