Elasticsearch query input plugin support for ES v7

[aymanrb]

Use Case

Querying logs stored in Elasticsearch version 7

Expected behavior

The connection works to Elasticsearch v7 and the query is executed

Actual behavior

I get the following error:

E! [inputs.elasticsearch_query] E! error connecting to elasticsearch: elasticsearch version 7.17.4 not supported (currently supported versions are 5.x and 6.x)

Additional info

No response

[powersj]

next steps: look into what is required to support v7

[aymanrb]

Since (un)fortunately the data source I want to query is in Elasticsearch 7 and would like to have telegraf support it so we keep the tool we use for collecting all metrics.

I tried to fiddle around quickly with it today and could get it to run against my datasource using the changes in this commit with the following configs:

[[inputs.elasticsearch_query]]
 urls = ["https://127.0.0.1:9200"]
 username = "{{UsernameHere}}"
 password = "{{PasswordHere}}"
 insecure_skip_verify = true
 enable_sniffer = false
 [[inputs.elasticsearch_query.aggregation]]
    measurement_name = "response_time"
    index = "nginx.api-logs*"
    date_field = "@timestamp"
    query_period = "5m"
    filter_query = "(nginx_http_host.keyword: {{HostHere}})"
    include_missing_tag = true
    missing_tag_value = "null"
    metric_fields = ["nginx_request_time"]
    metric_function = "avg"

I am not sure though if the changes I made to aggregation_query.go were needed actually for elasticsearch 7 or if is it more related to my index's mapping structure. I couldn't get the tests to run to try it with the test data/index provided on ES7, and once that's validated not sure also on how to make the plugin backward compatible with ES5 and ES6.

Any updates on the topic?

[powersj]

Thanks for taking a stab at this.

elastic5 "github.com/olivere/elastic/v7"

I do not think this will work, per this response and the readme which states the API version of the library needs to be the same as the version of the target elasticsearch server. I think we are currently getting lucky with using v5 with the v5+v6 server.

I think a PR to resolve this would need to add a config option to state which version of elasticsearch to target, with v5 as the default. And then based on that config option, create the correct client.

@srebhan thoughts?

[srebhan]

@powersj and @aymanrb there are multiple issues I see here. First of all, as @powersj said, you should additionally import

elastic7 "github.com/olivere/elastic/v7"

and not replace the existing client. While I'm not 100% sure about the compatibility I guess there is a reason why we see client versions matching server versions... The second issue I see is that github.com/olivere/elastic is deprecated according to the GitHub page and you should use github.com/elastic/go-elasticsearch instead. Previous naming scheme also applies there.

So given that we need to use a matching client version and the required migration of libraries, we should

1. Import all versions of the client we want to support

    elastic5 "github.com/elastic/go-elasticsearch/v5"
    elastic6 "github.com/elastic/go-elasticsearch/v6"
    elastic7 "github.com/elastic/go-elasticsearch/v7"
    elastic8 "github.com/elastic/go-elasticsearch/v8"

2. Define an interface elasticClient required to query the data
3. Implement instances matching the interface in 2. for each version v5, v6, v7, v8
4. Use elastic5 to query the server version and create a client instance matching this version (see 3)
5. Change the plugin query code to use the interface of 2
6. Extend integration tests to test against all major versions of elasticsearch

What do you think?

[aymanrb]

Yes sure, I definitely didn't mean we should remove the older version clients (was curious already about how did v5 work for v6 servers, yes absolutely we need to keep the support for all versions).

What I have in that commit was more of a quick check on what's needed (not an actual implementation, I even left the import alias as elastic5 to reduce the PoC changes), and as I found out it's not only the client package version that needs to be changed, I had to do changes to the aggregation_query.go too, and since I couldn't get the tests to work locally (it times out while waiting for the container to be ready) I couldn't verify if the changes were more of a new server response changes or something wrong with the index I am running my tests on.

What @srebhan suggested here sounds like a good plan (assuming the query code doesn't need changes besides using the right client).

4. Use elastic5 to query the server version and create a client instance matching this version (see 3)

Maybe just instead of this point, we can actually use a config value as @powersj suggested avoiding creating a client, connecting to ElasticSearch, and querying with the wrong client version already just to destroy it and create the proper one?

but generally speaking, that should work yes, again with the assumption that the query implementation doesn't require any adaptation between the different versions.

[srebhan]

Having an option also sounds good.

[aymanrb]

Ok and I guess we will need the aggregation query changes, unfortunately.

And generally speaking, the whole plugin needs to be rewritten (to some extent) if we want to use the official Elasticsearch client (elastic/go-elasticsearch) as suggested? The client is totally different from what olivere/elastic offered

[srebhan]

@aymanrb honestly speaking it causes me some headaches to add features based on a deprecated library. So even though it sounds like a big effort, I would rather prefer the rewrite. Maybe make it a elastic_query_v2 plugin...

What do you think @aymanrb and @powersj ?

[powersj]

If a user provides a PR with support for v7 I think we are in agreement that it should be with the new client and probably a new plugin.

[aymanrb]

I totally agree as well, especially since v8 anyway can't be supported by the old/deprecated client. So rewriting is inevitable on the very short term even. It's just that unfortunately I wouldn't have the time personally to tackle this in the meantime. So hopefully someone picks it up and gets the plugin's v2 on track.

[milank78git]

Hello, I want to ask, will there be a version of the Elasticsearch for Queries plugin available for ES 7.x? Thanks

[srebhan]

@milank78git someone would need to create a pull-request and add support... Help is appreciated!

[milank78git]

Unfortunately I can't program that well and I'm not from a big company :-(

I love the telegraph it's a shame great tool thanks

Dne út 19. 3. 2024 10:18 uživatel Sven Rebhan @.***> napsal:

@milank78git https://github.com/milank78git someone would need to create a pull-request and add support... Help is appreciated!

— Reply to this email directly, view it on GitHub https://github.com/influxdata/telegraf/issues/12099#issuecomment-2006453914, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOB5Y2SL4XH5BS5H5N76TU3YY77HBAVCNFSM6AAAAAARNXWVVOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBWGQ2TGOJRGQ . You are receiving this because you were mentioned.Message ID: @.***>