apm-server
apm-server copied to clipboard
self-instrumentation triggers mapping exceptions
start apm-server with self-instrumentation enabled
./apm-server -E instrumentation.enabled=true -e -d '*'
wait a minute or less and notice:
{"log.level":"error","@timestamp":"2022-08-09T13:37:42.589-0400","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":412},"message":"failed to index event (strict_dynamic_mapping_exception): mapping set to strict, dynamic introduction of [parent] within [process] is not allowed","service.name":"apm-server","trace.id":"58bdee8653d6445c590ad24451d57302","transaction.id":"58bdee8653d6445c","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-09T13:37:42.589-0400","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":430},"message":"bulk request completed: 2 indexed, 5 failed (0 exceeded capacity)","service.name":"apm-server","trace.id":"58bdee8653d6445c590ad24451d57302","transaction.id":"58bdee8653d6445c","ecs.version":"1.6.0"}
successes and failures continue to be logged after that.
Was this with 8.5.0-SNAPSHOT? Which commit of Kibana are you running? There was a change a couple of days ago to switch from process.ppid
to process.parent.pid
(https://github.com/elastic/apm-server/pull/8783). The latest 8.5.0 integration package in snapshot storage is up to date: https://github.com/elastic/package-storage/commit/4c2a9ccd072122c41405ba24efdc3f7644b50ecf
If Kibana is stale, then the bundled integration package would also be stale.
My kibana snapshot is older than that package so I suspect you're right and this is already fixed. I'll close this after confirming with a newer kibana build.
Relaunched the whole stack and still seeing the same with:
Kibana (image built: 2022-08-09 11:40:11 UTC):
Version: 8.5.0-SNAPSHOT
Branch: main
Build SHA: 6655eeff50554078e3f427828846399066cc29e3
Build number: 55383
Elastic Agent managed APM Server (image built: 2022-08-10 04:31:22 UTC):
Binary: 8.5.0-SNAPSHOT (build: dc5b1a2ec19c04e54f663be166e54f892d3c3385 at 2022-08-10 04:25:59 +0000 UTC)
The APM Server displaying the errors is running standalone from:
apm-server version 8.5.0 (amd64), libbeat 8.5.0 [32ace4565111ef7162bebe99a3a4940e4b31c96a built 2022-08-09 04:01:52 +0000 +0000]
How can the integration package version be determined?
How can the integration package version be determined?
It's not possible to directly check it, at the moment, as we replace the package version during development. I might look at adding the commit hash in the integration README.
The 8.5.0-SNAPSHOT images are currently bundling the 8.4.0 integration package, so that would explain the issue you're seeing. This will be fixed by https://github.com/elastic/kibana/issues/137759#issuecomment-1204191846
The integration package was updated, therefore this should no longer be an issue.