document apm package bundling with Fleet/Kibana

[simitt]

The question how the apm package is bundled with Fleet/Kibana and why there are no more releases of production versions of apm package available at the package registry is a continuous source of confusion.

We should document

• the flow how the bundled apm package gets updated for internal developers
• that the apm package is not expected to be available at the package registry in a production version as it is bundled with Kibana.

[axw]

the flow how the bundled apm package gets updated for internal developers

Packages that are bundled with Kibana are specified in https://github.com/elastic/kibana/blob/main/fleet_packages.json. There is a CI job (internal link) that runs periodically for each release branch, and updates this file.

The CI job does this by querying the EPR (Elastic Package Registry) for the latest package compatible with the Kibana version series (e.g. "what's the latest 'apm' package compatible with 8.7.x"). For automatically installing the package using the Fleet preconfiguration settings, EPR need only be available when building Kibana. There are some limitations in Fleet that currently require EPR to be available for certain other uses of the Fleet/Integrations apps, as noted in https://github.com/elastic/apm-server/issues/10361#issuecomment-1450166404.

At the time of writing, the apm package is the only one with some special flags defined in fleet_packages.json:

• forceAlignStackVersion causes the bundled package's version to be updated to be exactly the same as Kibana's
• allowSyncToPrerelease includes pre-release package versions when querying the latest package to bundle

We have the same reason for setting both of these attributes:

1. we consider the APM integration package to be part of the stack release
2. the stack release involves build promotion (i.e. creating multiple build candidates with the same final version, and then finally calling one of them the GA release)
3. integration package versions published to EPR must be both unique and immutable (conflicts with (2))

So our options would be to:

1. have an integration package versioning scheme independent of the stack
2. build unique package versions, and then update the version when bundling with Kibana

We're using approach (2) at the moment; the premise behind this was to avoid introducing any possibility of version skew between Kibana and the integration package, cause potential confusion for users, and impact supportability. It hasn't been the smoothest of sailing though; we still see issues in Fleet where these prerelease versions leak through to the UI.

Building and publishing of these integration packages is currently done in Jenkins: https://github.com/elastic/apm-server/blob/38d31a3aff8866eb4ed2b4b59f5880d8ea3eea8f/.ci/packaging.groovy#L138

[lucabelluccini]

Let's take the opportunity to tell how to install the package without having EPR connectivity nor a fully functional fleet (via API or Kibana settings). We have an internal KB about it (7e12b56e)

[simitt]

Holding off a bit with this, as there are currently conversations ongoing how to make the package installation a default.