opentelemetry-collector-contrib icon indicating copy to clipboard operation
opentelemetry-collector-contrib copied to clipboard
verified publisher icon open-telemetry

[exporter/elasticsearch] Bulk indexer error: an id must be provided if version type or value are set

Open lmssy opened this issue 1 year ago • 1 comments

Component(s)

exporter/elasticsearch

What happened?

Description

I'm trying to export trace & span to elasticsearch v7, I config collector just follow the document https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter but in collector's log report below error, there is no index created in elasticsearch. please help to check, thx.

image

Steps to Reproduce

Expected Result

Actual Result

Collector version

0.100.0

Environment information

Environment

OpenTelemetry Collector configuration

extensions:
  health_check:
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
exporters:
  file:
    path: /agent.json
  debug:
  elasticsearch/trace:
    endpoints: [http://my-elastich-host:9200]
    traces_index: trace_index
    user: xxx
    password: xxx
processors:
  batch/trace:
  memory_limiter/trace:
    limit_mib: 3000
    spike_limit_mib: 600
    check_interval: 5s
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: slow_calls
        type: latency
        latency:
          threshold_ms: 5000
      - name: error_otel_status
        type: status_code
        status_code:
          status_codes:
            - ERROR
      - name: error_http_status
        type: numeric_attribute
        numeric_attribute:
            key: http.status_code
            min_value: 500
      - name: keep_sampled_success
        type: and
        and:
          and_sub_policy:
            - name: drop_noisy_traces_url
              type: string_attribute
              string_attribute:
                key: http.url
                values:
                  - \/metrics
                  - opentelemetry\.proto
                  - favicon\.ico
                  - \/health
                enabled_regex_matching: true
                invert_match: true
            - name: keep_percentage
              type: probabilistic
              probabilistic:
                sampling_percentage: 100
service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter/trace,tail_sampling,batch/trace]
      exporters: [debug,file,elasticsearch/trace]

Log output

2024-05-21T13:48:18.897+0800	error	[email protected]/elasticsearch_bulk.go:150	Bulk indexer error: flush: [400 Bad Request] {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;2: an id must be provided if version type or value are set;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;2: an id must be provided if version type or value are set;"},"status":400}	{"kind": "exporter", "data_type": "traces", "name": "elasticsearch/trace"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/elasticsearch_bulk.go:150
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/[email protected]/esutil/bulk_indexer.go:324
2024-05-21T13:48:18.907+0800	error	[email protected]/elasticsearch_bulk.go:150	Bulk indexer error: flush: [400 Bad Request] {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;2: an id must be provided if version type or value are set;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;2: an id must be provided if version type or value are set;"},"status":400}	{"kind": "exporter", "data_type": "traces", "name": "elasticsearch/trace"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/elasticsearch_bulk.go:150
github.com/elastic/go-elasticsearch/v7/esutil.(*worker).flush
	github.com/elastic/go-elasticsearch/[email protected]/esutil/bulk_indexer.go:568
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/[email protected]/esutil/bulk_indexer.go:321
2024-05-21T13:48:18.907+0800	error	[email protected]/elasticsearch_bulk.go:150	Bulk indexer error: flush: [400 Bad Request] {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;2: an id must be provided if version type or value are set;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;2: an id must be provided if version type or value are set;"},"status":400}	{"kind": "exporter", "data_type": "traces", "name": "elasticsearch/trace"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/elasticsearch_bulk.go:150
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/[email protected]/esutil/bulk_indexer.go:324
2024-05-21T13:48:18.968+0800	error	[email protected]/elasticsearch_bulk.go:150	Bulk indexer error: flush: [400 Bad Request] {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;2: an id must be provided if version type or value are set;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;2: an id must be provided if version type or value are set;"},"status":400}	{"kind": "exporter", "data_type": "traces", "name": "elasticsearch/trace"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/elasticsearch_bulk.go:150
github.com/elastic/go-elasticsearch/v7/esutil.(*worker).flush
	github.com/elastic/go-elasticsearch/[email protected]/esutil/bulk_indexer.go:568
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/[email protected]/esutil/bulk_indexer.go:321
2024-05-21T13:48:18.968+0800	error	[email protected]/elasticsearch_bulk.go:150	Bulk indexer error: flush: [400 Bad Request] {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;2: an id must be provided if version type or value are set;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;2: an id must be provided if version type or value are set;"},"status":400}	{"kind": "exporter", "data_type": "traces", "name": "elasticsearch/trace"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/elasticsearch_bulk.go:150
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/[email protected]/esutil/bulk_indexer.go:324
2024-05-21T13:48:19.010+0800	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 1}
2024-05-21T13:48:21.016+0800	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 2, "spans": 2}
2024-05-21T13:48:24.026+0800	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 1}
2024-05-21T13:48:31.045+0800	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 1}

Additional context

No response

lmssy avatar May 21 '24 05:05 lmssy

Pinging code owners:

  • exporter/elasticsearch: @JaredTan95 @ycombinator @carsonip

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] avatar May 21 '24 05:05 github-actions[bot]

anyone can take a look for this issue?

lmssy avatar May 29 '24 07:05 lmssy

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

  • exporter/elasticsearch: @JaredTan95 @ycombinator @carsonip

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] avatar Jul 29 '24 03:07 github-actions[bot]

does anyone knows the solution ?

medorock25 avatar Aug 23 '24 09:08 medorock25

@lmssy Have you solved this issue ?

medorock25 avatar Sep 02 '24 12:09 medorock25

Sorry for the late reply. I am unable to reproduce the issue with v7.17.23 Elasticsearch and v0.108.0 elasticsearchexporter with a simpler config:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
exporters:
  debug:
  elasticsearch/trace:
    endpoints: [https://redacted.elastic-cloud.com]
    traces_index: trace_index
    user: redacted
    password: redacted
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [debug,elasticsearch/trace]

Questions

  • Do you get the same error without processors memory_limiter/trace,tail_sampling,batch/trace?
  • What kind of otlp payload are you sending to the collector? Does this happen with a very simple test span?
  • Which exact v7 version are you using?
  • Are there 0 documents indexed into Elasticsearch despite the error logs? Do you notice a pattern within the dropped documents, if there are any?
  • Are you somehow assigning _id or _version to the documents?

carsonip avatar Sep 02 '24 17:09 carsonip

I encountered the same error when using ES V7.3.2, but upgraded ES to V7.17.24 and restored it to normal

t673afa avatar Oct 09 '24 14:10 t673afa

I managed to reproduce this. It is because in version ES 7.3.2, bulk API create action requires an ID. It is not required after https://github.com/elastic/elasticsearch/pull/45806 which was released in 7.5.0. As these are all very old versions and are no longer supported, elasticsearchexporter is not going to add code to handle these cases. Please update your Elasticsearch to fix these errors.

This issue is good to close.

carsonip avatar Nov 20 '24 14:11 carsonip