openshift
Log-2769: Support Elasticsearch 8.x as an output using vector
Description
Summary:
- Added optional
Elasticsearch.Versionfield to types. - Added
suppress_type_name = trueif forwarding toElasticsearch 7.xor higher. - Can forward both to
defaultand externalElasticsearchstore at the same time with aClusterLogForwardercustom resource.
Details:
This PR addresses an issue with forwarding logs using vector to Elasticsearch 8.x (ES). Forwarding logs to ES 8.x produced the following error in the logs: Action/metadata line [1] contains an unknown parameter [_type].
To fix the error, in the vector.toml config file, [sinks - <ES>] section, suppress_type_name = true needs to be added for ES versions 7.x and higher. However, we cannot suppress the type name for ES 6.x, which is our internally managed ES.
To distinguish between ES versions, in the output_types.go file I added the Version field under Elasticsearch. This field is optional and will default to the latest ES version (right now that is 8.x) if not specified by the end user when defining a ClusterLogForwarder custom resource. The internal default store will be ES 6.x.
With this field, the suppress_type_name = true will only be added to vector.toml ES sink configurations using version 7.x or higher.
/cc @cahartma
Links
- https://issues.redhat.com/browse/LOG-2769
Hi @Clee2691. Thanks for your PR.
I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
![openshift-ci[bot] avatar](https://avatars.githubusercontent.com/in/91280?v=4 "Published by a pub.dev verified publisher"){kind=small}
/test e2e
Re-running because of error in e2e-gather-must-gather:
INFO[2022-07-05T22:08:02Z] Running must-gather...
error: yaml: line 1129: did not find expected key
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"k8s.io/test-infra/prow/entrypoint/run.go:80","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2022-07-05T22:07:58Z"}
error: failed to execute wrapped command: exit status 1
/test e2e
Re-running again as test timed out this time but all tests passed.
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Clee2691, jcantrill
The full list of commands accepted by this bot can be found here.
The pull request process is described here
- ~~OWNERS~~ [jcantrill]
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
/retest-required
Remaining retests: 2 against base HEAD fbab40fe6fefd79312624bbf755afd4a49c47d6a and 8 for PR HEAD b7338d44409a68afbdbe35ef990d88f03dbc7643 in total
/retest-required
Remaining retests: 1 against base HEAD fbab40fe6fefd79312624bbf755afd4a49c47d6a and 7 for PR HEAD b7338d44409a68afbdbe35ef990d88f03dbc7643 in total
/retest-required
Remaining retests: 0 against base HEAD fbab40fe6fefd79312624bbf755afd4a49c47d6a and 6 for PR HEAD b7338d44409a68afbdbe35ef990d88f03dbc7643 in total
/retest-required
Remaining retests: 2 against base HEAD de55678ca66d445758c298f2202466addcd101f6 and 5 for PR HEAD b7338d44409a68afbdbe35ef990d88f03dbc7643 in total
/retest-required
Remaining retests: 1 against base HEAD de55678ca66d445758c298f2202466addcd101f6 and 4 for PR HEAD b7338d44409a68afbdbe35ef990d88f03dbc7643 in total
/retest-required
Remaining retests: 0 against base HEAD de55678ca66d445758c298f2202466addcd101f6 and 3 for PR HEAD b7338d44409a68afbdbe35ef990d88f03dbc7643 in total
/retest-required
Remaining retests: 2 against base HEAD 3f272124c5faacec7ec8887842b260804b04265a and 2 for PR HEAD b7338d44409a68afbdbe35ef990d88f03dbc7643 in total
/retest-required
Remaining retests: 1 against base HEAD 3f272124c5faacec7ec8887842b260804b04265a and 1 for PR HEAD b7338d44409a68afbdbe35ef990d88f03dbc7643 in total
/retest-required
Remaining retests: 0 against base HEAD 3f272124c5faacec7ec8887842b260804b04265a and 0 for PR HEAD b7338d44409a68afbdbe35ef990d88f03dbc7643 in total
/hold
Revision b7338d44409a68afbdbe35ef990d88f03dbc7643 was retested 9 times: holding
/retest-required
Remaining retests: 2 against base HEAD 7b576316155449d364796e2b850c0fb9bcd1bd3f and 8 for PR HEAD fea038961e3d11098dd7c279789bbbea08665b7a in total
/retest-required
Remaining retests: 1 against base HEAD 7b576316155449d364796e2b850c0fb9bcd1bd3f and 7 for PR HEAD fea038961e3d11098dd7c279789bbbea08665b7a in total
/retest-required
Remaining retests: 0 against base HEAD 7b576316155449d364796e2b850c0fb9bcd1bd3f and 6 for PR HEAD fea038961e3d11098dd7c279789bbbea08665b7a in total