`otlp_config` has no effect

[zarbis]

Describe the bug I'm trying to utilize otlp_config to shift some attributes I was previously dropping into newly introduced Structured metadata. My attempts were ineffective and it's hard to distinguish indexed label and non-indexed attribute in Explore UI, so I decided to test radical removal of attributes with the following config.

limits_config:
  otlp_config:
    resource_attributes:
      attributes_config:
      - action: drop
        attributes:
        - k8s.pod.uid
        - k8s.pod.name
        - k8s.deployment.name
    log_attributes:
      - action: drop
        regex: .*
    scope_attributes:
      - action: drop
        regex: .*

This still resulted in no changes in observed Fields in Explore UI. So I've enabled debug logging on OTel Collector to see the data I'm sending to Loki. Here is an example of log record produced by debug exporter:

ResourceLog #15
Resource SchemaURL: 
Resource attributes:
     -> app.name: Str(REDACTED)
     -> build.id: Str(REDACTED)
     -> k8s.namespace.name: Str(REDACTED)
     -> k8s.node.name: Str(REDACTED)
     -> k8s.pod.name: Str(REDACTED)
     -> k8s.pod.uid: Str(REDACTED)
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2024-04-24 17:36:23.589775336 +0000 UTC
Timestamp: 2024-04-24 17:36:23.420370489 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(REDACTED)
Attributes:
     -> pod: Str(REDACTED)
     -> container: Str(REDACTED)
     -> namespace: Str(REDACTED)
     -> cluster: Str(REDACTED)
     -> alerting_tier: Str(REDACTED)
Trace ID: 
Span ID: 
Flags: 0

And here is the list if observed Fields So it is still exactly a combination of resource_attributes and log_attributes plus auto-discovered level and service_name.

Additionally I've confirmed, that k8s_pod_name is indexed label while k8s_pod_uid is non-indexed field, which follows default behavior. And mere presence of pod, container, namespace, cluster and alerting_tier makes me think that otlp_config is completely ineffective.

In ingester's debug log I see lines like those:

caller=flush.go:176 component=ingester msg="flushing stream" user=<...> fp=65e0db5cd9e011cf immediate=false num_chunks=1 labels="{k8s_deployment_name=\"<...>\", k8s_namespace_name=\"<...>\", k8s_pod_name=\"<...>\", service_name=\"<...>\"}"

so it somehow still produces those labels I'm trying to drop.

Expected behavior I expect majority of Fields to be absent in Explore, plus reduced resource consumption by Loki.

Environment:

• Infrastructure: Kubernetes (EKS 1.28)
• Deployment tool: Helm chart 6.3.4

[JStickler]

@zarbis, updating your configuration is only one part of the implementation, how are you Attaching the structured metatdata to your log lines?

[zarbis]

@JStickler I expected it to be ingested by OTLP endpoint naturally due to how it's documented:

Structured Metadata: Anything which can’t be stored in Index labels and LogLine would be stored as Structured Metadata.

[JStickler]

So you're using the OpenTelemetry Collector, which means you must use the otlphttp exporter, which does not support structured metadata.

The docs need to be updated, as we have two OLTP solutions.

[zarbis]

Indeed I do. The entire Ingesting logs to Loki using OpenTelemetry Collector doc reads like:

1. run otlphttp exporter in your OTEL Collector
2. customize otlp_config in Loki (optional)

If otlphttp indeed is incompatible with structured metadata, why does the entire doc is literally the opposite story? Sounds really confusing...

Just in case, this is relevant part of my Collector config (I run entire LGTM stack through it so showing only logs part of it):

config:
  receivers:
        otlp:
          protocols:
            http: ...
  exporters:
    otlphttp/loki:
        endpoint: http://loki-gateway/otlp
        auth: ...
        sending_queue: ...
    loki:
        endpoint: http://loki-gateway/loki/api/v1/push
        auth: ...
        sending_queue: ...

  service:
    pipelines:
      logs:
        receivers:
        - otlp
        processors:
        - memory_limiter
        - attributes/loki
        - resource/loki
        - batch
        exporters:
        - debug/normal
        # - otlphttp/loki
        - loki

I've tried running otlphttp/loki while also disabling attributes/loki and resource/loki since I supposed they won't be needed in native OTLP mode.

[sandeepsukhani]

Hey @zarbis, sorry for the inconvenience and thanks for reporting the issue. It seems there is some issue with applying the otlp_config set in the global limits_config, i.e. in the overall Loki config file. It is working fine as expected when set as per tenant config. While I get this fixed, can you please give it a try with the per-tenant config and use that to avoid getting blocked? I will update here when the fix gets merged.

[zarbis]

@sandeepsukhani I've moved otlp_config under runtimeConfig.overrides.<org> and it kinda works, but not really:

otlp_config:
  resource_attributes:
    attributes_config:
    - action: drop
      attributes:
      - k8s.pod.name
      - k8s.deployment.name
  log_attributes:
    - action: drop
      regex: .*
  scope_attributes:
    - action: drop
      regex: .*

log_attributes are dropped, scope_attributes I've had none to begin with, but resource_attributes are still present. I'm not sure whether the difference in different treatment of resource_attributes and other ones, or regex vs attributes list, but labels I've tried to drop are still there:

logcli --org-id <org> series --from="$(date --utc --iso-8601=seconds --date='5 minutes ago')" --analyze-labels '{}'
2024/04/30 21:08:26 http://localhost:3100/loki/api/v1/series?end=1714500506642625448&match=%7B%7D&start=1714500206000000000
Total Streams:  1653
Unique Labels:  8

Label Name                 Unique Values  Found In Streams
k8s_pod_name               1608           1626
k8s_deployment_name        268            745
k8s_namespace_name         40             1626
__stream_shard__           5              23
exporter                   1              21
service_name               1              1653