opentelemetry-collector-contrib icon indicating copy to clipboard operation
opentelemetry-collector-contrib copied to clipboard

Collector not working when k8sattributes in use

Open clintonb opened this issue 4 months ago • 1 comments

Component(s)

processor/k8sattributes

What happened?

Description

I am trying to add k8sattributes to a gateway collector, but the collector and health check are not functioning. The collector appears to start, but refuses connections on the receiving ports. The health check endpoint returns a 503 with {"status":"Server not available","upSince":"0001-01-01T00:00:00Z","uptime":""}.

Steps to Reproduce

  1. Build an image with the configuration below.
  2. Run it.

Expected Result

  1. Collector responds to health checks at curl -v http://localhost:13133.
  2. Received traces include k8s attributes.

Actual Result

The collector never becomes healthy, and does not accept any signals.

Collector version

0.111.0

Environment information

Environment

OS: macOS 15.0.1 (Docker), and GKE Autopilot

OpenTelemetry Collector configuration

extensions:
  health_check:
    endpoint: "0.0.0.0:13133"

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"
        include_metadata: true

processors:
  k8sattributes:
    auth_type: "serviceAccount"
    extract:
      metadata:
#        - k8s.namespace.name
#        - k8s.deployment.name
#        - k8s.statefulset.name
#        - k8s.daemonset.name
#        - k8s.cronjob.name
#        - k8s.job.name
#        - k8s.node.name
        - k8s.pod.name
#        - k8s.pod.uid
#        - k8s.pod.start_time
    passthrough: false
#    pod_association:
#      - sources:
#          - from: resource_attribute
#            name: k8s.pod.ip
#      - sources:
#          - from: resource_attribute
#            name: k8s.pod.uid
#      - sources:
#          - from: connection

exporters:
  # NOTE: Add this to the list of pipeline exporters to see the collector's debug logs
  debug:
    verbosity: detailed
service:
  extensions: [ health_check ]
  # https://opentelemetry.io/docs/collector/configuration/#telemetry
  telemetry:
    # This controls log verbosity of the collector itself.
    logs:
      encoding: json
      level: "debug"
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ k8sattributes ]
      exporters: [ debug ]

Log output

otel-collector-1  | {"level":"info","ts":1729289848.7740626,"caller":"[email protected]/service.go:136","msg":"Setting up own telemetry..."}
otel-collector-1  | {"level":"info","ts":1729289848.7742362,"caller":"telemetry/metrics.go:70","msg":"Serving metrics","address":"localhost:8888","metrics level":"Normal"}
otel-collector-1  | {"level":"info","ts":1729289848.7743702,"caller":"builders/builders.go:26","msg":"Development component. May change in the future.","kind":"exporter","data_type":"traces","name":"debug"}
otel-collector-1  | {"level":"debug","ts":1729289848.774658,"caller":"builders/builders.go:24","msg":"Beta component. May change in the future.","kind":"processor","name":"k8sattributes","pipeline":"traces"}
otel-collector-1  | {"level":"debug","ts":1729289848.774745,"caller":"builders/builders.go:24","msg":"Stable component.","kind":"receiver","name":"otlp","data_type":"traces"}
otel-collector-1  | {"level":"debug","ts":1729289848.7748547,"caller":"builders/extension.go:48","msg":"Beta component. May change in the future.","kind":"extension","name":"health_check"}
otel-collector-1  | {"level":"info","ts":1729289848.7753859,"caller":"[email protected]/service.go:208","msg":"Starting otelcol-contrib...","Version":"0.111.0","NumCPU":16}
otel-collector-1  | {"level":"info","ts":1729289848.775412,"caller":"extensions/extensions.go:39","msg":"Starting extensions..."}
otel-collector-1  | {"level":"info","ts":1729289848.775442,"caller":"extensions/extensions.go:42","msg":"Extension is starting...","kind":"extension","name":"health_check"}
otel-collector-1  | {"level":"info","ts":1729289848.7754776,"caller":"[email protected]/healthcheckextension.go:33","msg":"Starting health_check extension","kind":"extension","name":"health_check","config":{"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"ReadTimeout":0,"ReadHeaderTimeout":0,"WriteTimeout":0,"IdleTimeout":0,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
otel-collector-1  | {"level":"warn","ts":1729289848.7757652,"caller":"[email protected]/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.","kind":"extension","name":"health_check","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
otel-collector-1  | {"level":"info","ts":1729289848.7758508,"caller":"extensions/extensions.go:59","msg":"Extension started.","kind":"extension","name":"health_check"}


### Additional context

I run `curl -v http://localhost:13133` to check if the collector is healthy.

clintonb avatar Oct 18 '24 22:10 clintonb