apm-server
apm-server copied to clipboard
elastic
`log.level` isn't mapped or populated for error messages
APM Server version (apm-server version): All
Description of the problem including expected versus actual behavior:
APM Errors don't currently include log.level when documents are indexed in Elasticsearch. This makes these documents awkward to consume and filter as a user since the logs cannot be included in searches where log.level:error.
Steps to reproduce:
- Capture an error event in an instrumented APM Application.
- Inspect the document and you'll see that
log.levelis used
Provide logs (if relevant):
Will is be possible to provide some screenshots please? I don't see any logs provided in the description.
@rubvs Here's some APM error logs:
They contain error.log.level and error.log.message instead of the ECS log.level and message.
Having the same issue! Waiting for reply, for now my workaround is to have ingest pipeline for that, which also works a bit weird (not adding log level on Stream view).
Waiting for reply
@shevkaboss and @smith we are already working on a fix and intend to include it in an upcoming minor version release.
@rubvs thanks for quick reply.
Just wondering, is that possible also to show error level on Stream view? For traces, system Message field will be like "[dataSet.platform] [log.level] message". I would love to see that as "[error] message".
@shevkaboss we're going to be making some updates to the log stream view soon that might help you with this. Can you set Logs Explorer up to show this how you want?
have the same exact behavior in current latest elastic cloud deployment and otel collector streaming of logs from eks cluster services.
reproducible by: install helm chart of opentelemetry collector (any latest version, doesn't matter much since the issue is in the data stream index template mismatch, the 'log.level' does not exist in template mappings and dynamic=false) set otlp from some app to be collected, stream it to elastic with exporter, and 'log.level' field will not be searchable in the _index_template/logs-apm.error because one of the component templates responsible for explicit field mappings _component_template/logs-apm.error@package has field named 'error.log.level' (nested under error) but does not have 'log.level' specified (like for example the _component_template/logs-apm.app@package has).
so the field is searchable in the logs-apm.app indexes and not searchable in logs-apm.error indexes.
I think it's worth to reopen this bug report? the index template we use is the one 'managed by fleet' so we did not do any customization of template on our side except adding lifecycle policy using the recommended method in adding to GET _component_template/logs-apm.error@custom.
For now I will resolve this on our side by adding the require mapping for 'log.level' in the custom component-template as well, but it seems like a bug because many scenarios involve the field 'log.level' , especially if the otel collector streams it that way, I think it's a popular one that has to be supported out of the box?
what leads me to believe it's a bug (a missing field in default templates provided by apm) is that _component_template/logs-apm.app@package actually does have the correctly nested 'log.level' in it. but _component_template/logs-apm.error@package doesn't have.
'log.level' field will not be searchable in the _index_template/logs-apm.error because one of the component templates responsible for explicit field mappings _component_template/logs-apm.error@package has field named 'error.log.level' (nested under error) but does not have 'log.level' specified (like for example the _component_template/logs-apm.app@package has).
The component template [email protected] does define log.level as keyword by default, see https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/apm-data/src/main/resources/component-templates/logs-apm.error%40mappings.yaml#L10-L11
If you are running into further issues, feel free to raise questions or topics in the discuss forum.
'log.level' field will not be searchable in the _index_template/logs-apm.error because one of the component templates responsible for explicit field mappings _component_template/logs-apm.error@package has field named 'error.log.level' (nested under error) but does not have 'log.level' specified (like for example the _component_template/logs-apm.app@package has).
The component template
[email protected]does definelog.levelaskeywordby default, see https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/apm-data/src/main/resources/component-templates/logs-apm.error%40mappings.yaml#L10-L11If you are running into further issues, feel free to raise questions or topics in the discuss forum.
thanks for your reply!
is it the file responsible for populating the _component_template/logs-apm.error@package in current elastic cloud?
in both of our clusters, which are not modified from the moment we created them (one is from 2-3 months ago and another is from 1 month ago, so fairly recent clusters just created in elastic cloud console, with no modifications to those components) it looks like this (maybe I am missing something but I think those are made by elastic apm and not by our otel collector that streams data into cluster?):
there is no top level 'log' key in our case, so we possibly refer to different templates, which ones does the yaml file initialize? is it the source for _component_template/logs-apm.error@package ?
{
"component_templates": [
{
"name": "logs-apm.error@package",
"component_template": {
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "logs-apm.error_logs-default_policy"
},
"default_pipeline": "logs-apm.error-8.14.2",
"mapping": {
"total_fields": {
"limit": "1000"
}
}
}
},
"mappings": {
"dynamic": false,
"dynamic_templates": [
{
"http.request.env": {
"path_match": "http.request.env.*",
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
},
{
"http.request.cookies": {
"path_match": "http.request.cookies.*",
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
},
{
"http.request.headers": {
"path_match": "http.request.headers.*",
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
},
{
"http.response.headers": {
"path_match": "http.response.headers.*",
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
},
{
"labels": {
"path_match": "labels.*",
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
},
{
"numeric_labels": {
"path_match": "numeric_labels.*",
"mapping": {
"scaling_factor": 1000000,
"type": "scaled_float"
},
"match_mapping_type": "*"
}
}
],
"properties": {
"container": {
"properties": {
"id": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"kubernetes": {
"properties": {
"node": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"pod": {
"properties": {
"uid": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"namespace": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"parent": {
"properties": {
"id": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"agent": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"activation_method": {
"index": false,
"type": "keyword"
},
"ephemeral_id": {
"ignore_above": 1024,
"type": "keyword"
},
"version": {
"index": false,
"type": "keyword"
}
}
},
"session": {
"properties": {
"id": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"destination": {
"properties": {
"address": {
"ignore_above": 1024,
"type": "keyword"
},
"port": {
"type": "long"
},
"ip": {
"type": "ip"
}
}
},
"source": {
"properties": {
"nat": {
"properties": {
"port": {
"type": "long"
},
"ip": {
"type": "ip"
}
}
},
"port": {
"type": "long"
},
"domain": {
"ignore_above": 1024,
"type": "keyword"
},
"ip": {
"type": "ip"
}
}
},
"error": {
"properties": {
"exception": {
"properties": {
"code": {
"ignore_above": 1024,
"type": "keyword"
},
"stacktrace": {
"type": "flattened"
},
"handled": {
"type": "boolean"
},
"module": {
"ignore_above": 1024,
"type": "keyword"
},
"attributes": {
"type": "flattened"
},
"message": {
"type": "text"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"culprit": {
"ignore_above": 1024,
"type": "keyword"
},
"log": {
"properties": {
"stacktrace": {
"type": "flattened"
},
"level": {
"ignore_above": 1024,
"type": "keyword"
},
"logger_name": {
"ignore_above": 1024,
"type": "keyword"
},
"message": {
"type": "text"
},
"param_message": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"custom": {
"type": "flattened"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"stack_trace": {
"ignore_above": 1024,
"type": "wildcard",
"fields": {
"text": {
"type": "match_only_text"
}
}
},
"type": {
"ignore_above": 1024,
"type": "keyword"
},
"grouping_key": {
"ignore_above": 1024,
"type": "keyword"
},
"grouping_name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"network": {
"properties": {
"carrier": {
"properties": {
"mnc": {
"ignore_above": 1024,
"type": "keyword"
},
"icc": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"mcc": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"connection": {
"properties": {
"subtype": {
"ignore_above": 1024,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
},
"cloud": {
"properties": {
"availability_zone": {
"index": false,
"type": "keyword"
},
"instance": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"provider": {
"index": false,
"type": "keyword"
},
"machine": {
"properties": {
"type": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"service": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"project": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"region": {
"ignore_above": 1024,
"type": "keyword"
},
"account": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
},
"observer": {
"properties": {
"hostname": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
},
"version": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"trace": {
"properties": {
"id": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"numeric_labels": {
"dynamic": true,
"type": "object"
},
"host": {
"properties": {
"hostname": {
"ignore_above": 1024,
"type": "keyword"
},
"os": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
},
"type": {
"ignore_above": 1024,
"type": "keyword"
},
"version": {
"ignore_above": 1024,
"type": "keyword"
},
"platform": {
"index": false,
"type": "keyword"
},
"full": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
}
},
"ip": {
"type": "ip"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"architecture": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"client": {
"properties": {
"geo": {
"properties": {
"continent_name": {
"index": false,
"type": "keyword"
},
"region_iso_code": {
"index": false,
"type": "keyword"
},
"city_name": {
"index": false,
"type": "keyword"
},
"country_iso_code": {
"index": false,
"type": "keyword"
},
"country_name": {
"index": false,
"type": "keyword"
},
"location": {
"type": "geo_point"
},
"region_name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"port": {
"type": "long"
},
"domain": {
"ignore_above": 1024,
"type": "keyword"
},
"ip": {
"type": "ip"
}
}
},
"event": {
"properties": {
"kind": {
"ignore_above": 1024,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
},
"category": {
"ignore_above": 1024,
"type": "keyword"
},
"outcome": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"user_agent": {
"properties": {
"original": {
"index": false,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
},
"os": {
"properties": {
"kernel": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"index": false,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
},
"family": {
"ignore_above": 1024,
"type": "keyword"
},
"version": {
"ignore_above": 1024,
"type": "keyword"
},
"platform": {
"ignore_above": 1024,
"type": "keyword"
},
"full": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
}
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"device": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"version": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"timestamp": {
"properties": {
"us": {
"type": "long"
}
}
},
"process": {
"properties": {
"args": {
"ignore_above": 1024,
"type": "keyword"
},
"parent": {
"properties": {
"pid": {
"type": "long"
}
}
},
"pid": {
"type": "long"
},
"title": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
}
},
"message": {
"type": "match_only_text"
},
"processor": {
"properties": {
"event": {
"type": "constant_keyword",
"value": "error"
}
}
},
"url": {
"properties": {
"path": {
"ignore_above": 1024,
"type": "wildcard"
},
"fragment": {
"ignore_above": 1024,
"type": "keyword"
},
"original": {
"ignore_above": 1024,
"type": "wildcard",
"fields": {
"text": {
"type": "match_only_text"
}
}
},
"scheme": {
"ignore_above": 1024,
"type": "keyword"
},
"port": {
"type": "long"
},
"domain": {
"ignore_above": 1024,
"type": "keyword"
},
"query": {
"ignore_above": 1024,
"type": "keyword"
},
"full": {
"ignore_above": 1024,
"type": "wildcard",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
}
},
"labels": {
"dynamic": true,
"type": "object"
},
"@timestamp": {
"ignore_malformed": false,
"type": "date"
},
"data_stream": {
"properties": {
"namespace": {
"type": "constant_keyword"
},
"type": {
"type": "constant_keyword"
},
"dataset": {
"type": "constant_keyword"
}
}
},
"service": {
"properties": {
"node": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"environment": {
"ignore_above": 1024,
"type": "keyword"
},
"framework": {
"properties": {
"name": {
"index": false,
"type": "keyword"
},
"version": {
"index": false,
"type": "keyword"
}
}
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"runtime": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"version": {
"index": false,
"type": "keyword"
}
}
},
"language": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"version": {
"index": false,
"type": "keyword"
}
}
},
"version": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"http": {
"properties": {
"request": {
"dynamic": true,
"type": "object",
"properties": {
"referrer": {
"ignore_above": 1024,
"type": "keyword"
},
"headers": {
"dynamic": true,
"type": "object"
},
"method": {
"ignore_above": 1024,
"type": "keyword"
},
"body": {
"type": "flattened"
},
"env": {
"dynamic": true,
"type": "object"
},
"cookies": {
"dynamic": true,
"type": "object"
}
}
},
"response": {
"dynamic": true,
"type": "object",
"properties": {
"headers": {
"dynamic": true,
"type": "object"
},
"status_code": {
"type": "long"
},
"finished": {
"type": "boolean"
},
"headers_sent": {
"type": "boolean"
}
}
},
"version": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"device": {
"properties": {
"model": {
"properties": {
"identifier": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"manufacturer": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"user": {
"properties": {
"domain": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"email": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"transaction": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
},
"sampled": {
"type": "boolean"
}
}
},
"span": {
"properties": {
"id": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
}
},
"_meta": {
"package": {
"name": "apm"
},
"managed_by": "fleet",
"managed": true
}
}
}
]
}
8.14.2
You are using an older version. This has been fixed more recently, I recommend to upgrade to the latest available version (currently 8.17) for getting this fix.
Again, for any follow up questions, you should use the discuss forum. This is a closed issue for a bug that has been fixed.