Kibana does not remove index template for removed data streams on integrations upgrade

[endorama]

When we will migrate from apm integration package to apm-data ES plugin we will need to trigger a removal of index templates created by the integration, which by default are created with a higher priority than the one created by the ES plugin.

To ensure this steps works I run a test to verify if Kibana removes index template when upgrading an integration where installed data streams have been removed. I tested these steps:

1. setup a Es+Kibana stack and install apm integration package (tested with 8.15.0-SNAPSHOT but any current version should lead to same results)
2. check installed Index Templates
3. install a custom version of apm integration package where all data streams except one (traces - due to how elastic-package works is not possible to package an integration without at least 1 data stream; this is not relevant for this test but should be investigated to proceed further with the migration) have been removed
4. Verify if Index Templates for removed data streams have been removed.

The test showed that Index Templates are not removed. This is a blocker for https://github.com/elastic/apm-server/issues/11529 as templates from the plugin would be overridden by the integration package templates.

Assets difference between packages

Assets with normal integration package:

Assets with custom integration package:

Index templates before installing custom version

Index templates after installing custom version

[axw]

The test showed that Index Templates are not removed. This is a blocker for https://github.com/elastic/apm-server/issues/11529 as templates from the plugin would be overridden by the integration package templates.

We can unblock ourselves by increasing the priority of the templates in the apm-data plugin: https://github.com/elastic/apm-server/issues/11529#issuecomment-2119519247

Not ideal, but I think that's likely the most expedient approach. We should still go ahead with removing the data streams from the integration package, the existing templates just won't be removed. If need be, we could add some code to Kibana to remove them for APM specifically.

[endorama]

We moved forward with increasing the priorities in https://github.com/elastic/elasticsearch/pull/108885

I'd keep this issue open to track this Kibana behavior which may still affect the migration (es removing old templates is a manual step).

[simitt]

Let's work on removing the assets if possible to avoid customer confusion.