unexpected content-type error when exporting metrics and traced

[vishnujanardhanan]

Trying to setup and run adot collector with a java agent and push metrics and traces to opensearch via data prepper.

Error seen in the collector logs as below

{"level":"error","timestamp":"2022-09-14T15:57:40.069Z","caller":"exporterhelper/queued_retry.go:395","message":"Exporting failed. The error is not retryable. Dropping data.","kind":"exporter","data_type":"metrics","name":"otlp/data-prepper","error":"Permanent error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"","dropped_items":53,"stack":"go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:395\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:143\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:206\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:61"} {"level":"error","timestamp":"2022-09-14T15:58:40.069Z","caller":"exporterhelper/queued_retry.go:395","message":"Exporting failed. The error is not retryable. Dropping data.","kind":"exporter","data_type":"metrics","name":"otlp/data-prepper","error":"Permanent error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"","dropped_items":53,"stack":"go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:395\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:143\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:206\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:61"} <Aneurysm9: removed link potentially leaking credentials>

attaching the otel-config and data prepper config for reference. otel-config.txt

Running java application with aws adot agent on default configs.

Kindly help

[bryan-aguilar]

Can you please review the pipeline.txt file to ensure that it does not contain any valid credentials. If so please rotate those credentials immediately.

[vishnujanardhanan]

@bryan-aguilar - it doesn't. It is a phoenix environment which we spin up and destroy daily. So it is taken care off.

[rapphil]

can you share a redacted pipeline.txt again?

did you data-prepper pipeline contain both a otel_trace_source and a otel_metrics_source?

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been marked as stale for 30 days with no activity.