open-telemetry
How to remove duplicate & non relevant attributes from the Otlp Log record before it is exported to OTLP log collector destination
Question
How to remove duplicate & non relevant attributes from the Otlp Log record before it is exported to OTLP log collector destination
- Duplicate attributes : Open telemetry Log data model contains LogRecord.TraceId & LogRecord.SpainId attributes but same attributes also added LogRecord.attributes collection.
- non relevant attributes : LogRecord.attributes collection contains ActionId, ActionName, ConnectionId,
I tried adding my own LogProcessor using an derived implementation of BaseProcessor<LogRecord>. it does not work because LogRecord.attributes does not have these values when Otel library calls my implementation
public class DemoLogProcessor : BaseProcessor<LogRecord>{
public override void OnEnd(LogRecord data) {
// add logic to remove ActionId, ActionName, ConnectionId , TraceId and SpainId from
// LogRecord.Attributes Collection by creating new `IReadOnlyList<KeyValuePair<string, object?>` out of `LogRecord.Attributes`
}
}
Sample json log for reference
{
"resourceLogs": [
{
"resource": {
"attributes": [
{
"key": "service.name",
"value": {
"stringValue": "OtelLokiLogTest"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "7cf6b018-0330-41f6-9846-4d5d69a6571e"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "dotnet"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.6.0"
}
}
]
},
"scopeLogs": [
{
"logRecords": [
{
"timeUnixNano": "1701859790202535200",
"severityNumber": "SEVERITY_NUMBER_INFO",
"severityText": "Information",
"body": {
"stringValue": "Get Weatherforecast function called 2023-12-06 4:19:50 PM"
},
"attributes": [
{
"key": "date",
"value": {
"stringValue": "2023-12-06 4:19:50 PM"
}
},
{
"key": "{OriginalFormat}",
"value": {
"stringValue": "Get Weatherforecast function called {date}"
}
},
{
"key": "SpanId",
"value": {
"stringValue": "d840418c7f429c48"
}
},
{
"key": "TraceId",
"value": {
"stringValue": "a4ba025a812e1d889c062a8319f7e951"
}
},
{
"key": "ParentId",
"value": {
"stringValue": "0000000000000000"
}
},
{
"key": "ConnectionId",
"value": {
"stringValue": "0HMVM93BK0GTU"
}
},
{
"key": "RequestId",
"value": {
"stringValue": "0HMVM93BK0GTU:00000003"
}
},
{
"key": "RequestPath",
"value": {
"stringValue": "/weatherforecast"
}
},
{
"key": "ActionId",
"value": {
"stringValue": "9eb2e807-14ab-410a-99a8-995b556f9362"
}
},
{
"key": "ActionName",
"value": {
"stringValue": "OtelLokiLogTest.Controllers.WeatherForecastController.Get (OtelLokiLogTest)"
}
},
{
"key": "name",
"value": {
"stringValue": "Eloise Stracke"
}
},
{
"key": "age",
"value": {
"intValue": "82"
}
},
{
"key": "address",
"value": {
"stringValue": "Klingport"
}
}
],
"traceId": "pLoCWoEuHYicBiqDGffpUQ==",
"spanId": "2EBBjH9CnEg=",
"observedTimeUnixNano": "1701859790202535200"
}
]
}
]
}
]
}
Use Github Discussions.
non relevant attributes : LogRecord.attributes collection contains ActionId, ActionName, ConnectionId,
There are likely coming from ILogger scopes. The only option today to avoid these, is to not enable scopes support. (They are not enabled by default.)
There isn't any fine grained control over scopes today, and this is something that needs to be addressed.
non relevant attributes : LogRecord.attributes collection contains ActionId, ActionName, ConnectionId,
There are likely coming from ILogger scopes. The only option today to avoid these, is to not enable scopes support. (They are not enabled by default.)
There isn't any fine grained control over scopes today, and this is something that needs to be addressed.
Yes these are coming from log scope, but I need logger scope for application adds application specific attributes in logger scope. This is the only way to enrich logs with application specific attributes , need to have some extensibility to filter out the attributes that are not relevant for application .
List<KeyValuePair<string,object?>> logContext= new List<KeyValuePair<string,object?>
{
new KeyValuePair<string, object?>( "CustomerId","1234"),
new KeyValuePair<string, object?>( "OrderId","768"),
}
using (logger.BeginScope( logContext ))
{
logger.LogInformation("Processing credit card payment");
}
@cijothomas
Can't we provide some option in OpenTelemetryLoggerOptions that takes Scope filter as an option to retain or remove items ? Or some way by installing custom export processor
simple pseudo code here,
public class OpenTelemetryLoggerOptions
{
public Fun<bool ,...> ScopeFilter { get;set;}
}
Its definitely possible to offer some filtering ability, need to write up a proposal (or multiple), and gather feedback.
Could you create a more specific issue to tackle this? ILogger is adding more filtering capability in the next release, and it may be worth checking if scope based filtering should come as part of that or it should be solved at OTel level. A separate issue will help track this issue.
Its definitely possible to offer some filtering ability, need to write up a proposal (or multiple), and gather feedback. Could you create a more specific issue to tackle this?
ILoggeris adding more filtering capability in the next release, and it may be worth checking if scope based filtering should come as part of that or it should be solved at OTel level. A separate issue will help track this issue.
Done #5239