APM Integration index templates never created in docker/local environment

[dmathieu]

Describe the bug

Apologies if this is the wrong repository for reporting this issue. I am happy to recreate it elsewhere is necessary. When running Elasticsearch, Kibana and APM-Server locally, I cannot enable the APM integration, as the logs-apm.* and traces-apm.* index templates are never created.

To Reproduce

Run Elasticsearch and Kibana locally with docker:

docker network create elastic
docker run --name es01 --net elastic -p 9200:9200 -it docker.elastic.co/elasticsearch/elasticsearch:8.5.2
docker run --name kib-01 --net elastic -p 5601:5601 docker.elastic.co/kibana/kibana:8.5.2

Do the auth dance for kibana, and visit http://0.0.0.0:5601/app/home#/tutorial/apm to install the APM integration. Pick the "macOS" option (I suppose the windows option may show the same issue as well.

Follow the instructions to run apm-server locally, and start it (this isn't very obvious from the setup page either, especially due to the lack of proper SSL certificated in a docker setup, but that's another issue).

apm-server then shows the following errors (excerpt, but they're all the same, with all the index templates used by apm-server):

{"log.level":"error","@timestamp":"2022-12-08T10:04:15.425+0100","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [traces-apm.sampled] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [traces-apm.sampled] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-08T10:04:25.425+0100","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-apm.error] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-apm.error] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}

Checking for "APM Server Status" also checks for the presence of those index templates, so it doesn't detect it as running.

If I click on "Launch APM" anyway, there are more errors, still because the index templates don't exist.

Expected behavior

The documented way to run apm-server in a local/development environment for testing purposes should allow setting up the instrumentation.

[dmathieu]

It seems the 8.5.2 version of the APM isn't found in the registry? I noticed this in the kibana logs when I access the integration installation page:

[2022-12-13T11:17:30.807+00:00][ERROR][plugins.apm] RegistryResponseError: '404 Not Found' error response from package registry at https://epr.elastic.co/epr/apm/apm-8.5.2.zip
    at registryFetch (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/registry/requests.js:49:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12)

[simitt]

@kpollich could you take a look at this? It rather seems like an installation issue from the apm package from fleet than an apm-server issue.

[kpollich]

The APM package installs with the expected bundled version if 8.5.2 when I follow the steps to reproduce (copy/pasted the docker commands) here:

I see the expected index templates as well

Maybe I'm missing something but it appears to me that Fleet's installation of the APM integration worked as expected. The 404 we're seeing in the terminal is because apm-8.5.2 doesn't exist in the registry, and I think we're trying to request various assets like icons from the registry until https://github.com/elastic/kibana/issues/136617 is fixed.

@simitt let me know if I can help any other way, but I'm going to unassign this as it seems to me that Fleet's involvement here is working as intended.