[Fleet] Create UI for experimental indexing features

[kpollich]

Parent issue https://github.com/elastic/kibana/issues/132818

🗒️ Summary

In order to test upcoming Elasticsearch features and their feasibility, benefits, and functionality with integrations, we'd like to allow users to opt into several experimental indexing features:

• ~Synthetic _source - Saves on disk space by not storing the _source of each document~
  • (Handled separately in https://github.com/elastic/kibana/issues/140095 + https://github.com/elastic/kibana/pull/140132)
• doc-value-only fields - Allows querying certain field types without indexing them to save on disk space
• Time-series indexing (TSDB) - Optimize indexing for time-series data

To support opting into these options, we should expose a UI in the integration policy editor to allow opting in/out of each of these options. This settings will apply to all policies, even though we're presenting it at a single-data stream level. For this reason, we should make sure we present some kind of confirmation modal to alert the user that they're changing this setting for all policies with the given package/data stream.

Eventually, these settings will come from the package spec. For now, though, we need a way to allow users (probably internal ones) to test out these indexing features on various integrations/data streams. So, we're building out the "framework" for Fleet to support these settings, but handling the opt-in from our own UI instead of the package manifest files.

🛠️ Implementation

• [ ] Present a switch based UI (see designs below) for opting in/out of doc-value-only, and TSDB under "Advanced Options" for each data stream presented in the integration policy editor
  • [ ] Ensure that some language is present in a confirmation modal or elsewhere that indicates this change will apply to all policies for the current package/data stream - cc @dedemorton @dborodyansky as we'll need input on wording here
  • [ ] Each experimental feature should include a tooltip containing additional details/information and potentially a link to any documentation or other resources that are relevant - cc @dedemorton @dborodyansky again for the specific copy here (see designs below)
  • [ ] Persist these settings in the saved object representation of the package (epm-packages SO type, Installation model in Fleet UI codebase)
  • [ ] Saving these settings should trigger an update for the package's index template/component templates to enable the new settings
• [ ] Implement logic to apply these settings when generating an index template for a given package
  • This currently happens at "install time" for all packages
  • Reference https://github.com/elastic/integrations/blob/main/docs/how_to_test_new_indexing_features.md for how these settings work manually today
  • [ ] "Enrich" package mappings and to ensure they're compatible with each experimental feature
    • Note: We should only enrich @package component template mappings - NOT @custom ones. Users of @custom templates are "on their own" to opt into these features manually if they choose.
    • [ ] doc-value-only
      • [ ] Add index: false setting to every long and double mapping
    • [ ] TSDB
      • [ ] Add "index.mode": "time_series" index setting
      • [ ] Add "index.routing_path": ["..."] index setting ❓
      • [ ] Ensure time_series_dimension and time_series_metric is present on appropriate fields ❓
    • [ ] Reverse the "enrichment" process when a user opts out of experimental indexing options
      • e.g. if a user disabled the doc-value-only option, we need to go remove our index: false settings and update the index/component templates

👩‍💻 Code References

• epm-packages SO type definition: Link
• Mapping generation logic: Link

🎨 Designs

❓ Open Questions

• TSDB
  • How do we determine the index.routing_path value for a given data stream?
  • What do we do if time_series_dimension and time_series_metric settings aren't provided for the package? Right now, it seems the manual instructions expect a package to define these mappings. Does opting into TSDB make sense if a package hasn't set up the various dimension/metric values?
• Is the "mapping update" logic in the description above all valid and correct?

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

~Description here needs to be rewritten since we've captured synthetic source as its own issue in https://github.com/elastic/kibana/issues/140095~

Description has been updated to call out separate synthetic source implementation. Please reference the linked PR for some implementation decisions around things like saved object type schema updates, etc.

[kpollich]

This is closely related to https://github.com/elastic/kibana/issues/141211, but I think we can address wiring this functionality up to the package spec values for each data stream as a follow-up.

Building the UI and persistence out for each setting is enough work for this issue, and tacking on the functionality of honoring the package spec configuration for each data stream would be a bit much.

[kpollich]

Superceded by https://github.com/elastic/kibana/issues/132818