[Independent Agent Release] APM Manual Testing

[pierrehilbert]

Issue

Validate that the Independent Agent Release binary is not having side effect on running APM.

Context

To be able to fix critical issue faster, we worked on a new process allowing us to release a new Agent version without being coupled to Elastic Stack releases: this process is called Independent Agent Release.

In the context of the Independent Agent Release, we are packaging a new Elastic Agent for an existing patch version. This package contains opt-in and opt-out artifacts: APM is an opt-out artifact.

Being opt-out means that we are taking the last released version of the artifact for this patch version and not the last build version. In the example bellow, 8.14.3 artifacts are the opt-out artifacts:

elastic-agent-8.14.3+buildYYYYMMDD-darwin-aarch64
 └─ data
    └── elastic-agent-0219ac
        ├── components
        │   ├── apm-server          v8.14.3
        │   ├── endpoint-security   v8.14.4 commit 7dha76
        │   ├── filebeat            v8.14.4 commit 918bd3
        │   ├── fleet-server        v8.14.3
        │   ├── metricbeat          v8.14.4 commit 918bd3
        ├── elastic-agent           v8.14.4 commit 0219ac
        └── package.version         v8.14.3+buildYYYYMMDD

How to test

1. To be able to access the Independent Agent Release binaries (they are only available on staging environment for now), you first need to change the Agent Binary Source in Fleet UI: https://www.elastic.co/guide/en/fleet/current/fleet-settings.html#fleet-agent-binary-download-settings. You can use http://staging.elastic.co/independent-agent/8.14.3+build202408141002/downloads/ as the source URL.
2. Upgrade your previously installed Elastic Agent by manually choosing the 8.14.3+build202408141002 version in Fleet UI.
3. Verify the APM version is 8.14.3
4. Verify that APM is running as expected

[simitt]

@pierrehilbert will these EA versions be available as SNAPSHOTS on ESS so we can incorporate them into our automated smoke tests that we conduct for every version? We can schedule a brief manual test for our next iteration (Sep 9th-Sept 29th), but I'm eager to have this incorporated in automated testing, as my assumption is that there are not tests run by the EA team that concern APM Server functionality?

[pierrehilbert]

Unfortunately for now the Staging API is not reflecting the Independent Agent Release there but this is a topic we are working with the Release team as we currently only have it if we are start the process up to production. We currently have an opened issue to do it for our Agent upgrade tests and once it will be done, you could probably do something similar for APM.

[1pkg]

@pierrehilbert I followed the steps you shared to use the independent agent release binaries, and it seems to work as expected for the default agent policy for the agent hosted on the Elastic Cloud. However, for a new policy if I try to run the agent on my own host (either standalone or managed by fleet) I can't quite validate it, because the artifacts link that you shared http://staging.elastic.co/independent-agent/8.14.3+build202408141002/downloads/ doesn't have any relevant artifacts to install locally, for example http://staging.elastic.co/independent-agent/8.14.3+build202408141002/downloads/beats/elastic-agent/elastic-agent-8.14.3-linux-x86_64.tar.gz returns 404.

[pierrehilbert]

This artifact doesn't exist anymore as Elastic policy is cleaning staging artifacts after a defined period (a month if I remember correctly).

You should be able to use the "released" one from here: https://www.elastic.co/fr/downloads/past-releases/elastic-agent-8-14-3+build202408141002

[1pkg]

@pierrehilbert thank you for providing a link to past releases for this build. With these assets I was able to validate that APM Server runs as expected and our smoke tests are passing for Independent Agent Release build202408141002. All looks good in all scenarios: the standalone agent, the agent enrolled via fleet and the agent hosted on the Elastic Cloud.

[pierrehilbert]

Thanks @1pkg for running those manuals tests.