Automatically handle `TypeName` parameter for different versions of Elasticsearch (<7, 7 or higher)

[nenadvicentic]

trafficstars

What issue does this PR address?

• Changes default TypeName behavior to fit compatibility with Elasticsearch version 7, 8 and higher, instead of 6 or lower.
• When options ElasticsearchSinkOptions.DetectElasticsearchVersion is set to true handles automatically compatibility of TypeName parameter (_type in JSON body) against Elasticsearch version 7 or higher, as well version 6 or lover.

Does this PR introduce a breaking change? Yes. Default behavior, settings TypeName="logevent" unless TypeName has been set to null explicitly, has changed. TypeName remains null, if it has not been set manually. This fits with behavior expected by Elasticsearch version 7 or higher.

Please check if the PR fulfills these requirements

• [x] The commit follows our guidelines
• [x] Unit Tests for the changes have been added (for bug fixes / features)

[nenadvicentic]

@mivano @jonasmidstrup - Hi guys, when you have time, can you take a look into the PR?

Technically, this is going in a good direction, but I think it needs a few more things to be better rounded update.

1. Index Template for version Elasticserach v8
2. Maybe more clear option name to describe the behavior, instead of already existing DetectElasticsearchVersion.
3. To properly name and categorize unit and integration tests I've added.

Regarding unit-tests,, I added two of those the way I'm approaching naming, etc. One is, what I consider proper unit-test, another is actually integration test, but does not fit with logic of previous integration tests in this project. So, there I could use some guidance on conventions/intentions.

[jonasmidstrup]

@nenadvicentic Think it looks good 👍🏻 Have you tried it with different Elasticsearch versions?

I think it would be great to have an actual index template for v8.

[nenadvicentic]

@jonasmidstrup I tried this on Elasticsearch versions 7 and 8.

Regarding template for v8, I will take a look in next days, if I can figure out how to make working template, but this is a new topic for me.

[nenadvicentic]

@jonasmidstrup @mivano Guys, I am done with changes concerning this task. Please take a look when you have time.

Most important changes are:

• DetectElasticsearchVersion is now set to true by default. This helps determine how to handle TypeName across different major versions of elasticsearch and it also helps deciding which URL to use when uploading index template (v8 has new endpoint for this: _index_template vs old template.

• TypeName is null by default (although it is also touched once Elasticsearch version is detected).

• Internal class ElasticsearchVersionManager has been added, mainly to handle situations where detection of version fails or when it is disabled. In the case of fallback, sink will assume "default" version 7.

  Reasoning to use version 7 as a default is that Elasticsearch.NET client version 7.15.2 supports major versions of Elasticsearch server from 6 to 8 and that version 8 has "minimum_wire_compatibility_version" : "7.17.0" and "minimum_index_compatibility_version" : "7.0.0", so version 7 has broadest compatibility at the moment.

Other changes:

• Nuget pacakges have been updated (except for the Elasticsearch integration-tests related packages)
• Most of the ElasticserachSink functionality has been moved into internal BatchedElasticsearchSink class that inherits from IBatchedLogEventSink, so it complies with new recommended way of integration with PeriodicBatchingSink and we don't use obsolete constructors.
• ConnectionStub was moved out of ElasticsearchSinkTestsBase and extended. Both are now in /Stubs subfolder. Newer versions of Elasticsearch.NET client are now using "pre-flight" request to determine if endpoint is Elasticsearch and if it is indeed between 6 and 8. ConnectionStub had to accommodate for that.
• Unit tests have been fixed/added accordingly.

I have tested this version of code against Elasticsearch 7 and 8, using minimal configuration:

  "Serilog": {
    "Using": [ "Serilog.Sinks.Elasticsearch" ],
    "MinimumLevel": "Information",
    "WriteTo": [
      {
        "Name": "Elasticsearch",
        "Args": {
          "nodeUris": "http://iutlog1:9200"
        }
      }
    ]
  }

Regarding Serilog.Sinks.Elasticsearch.IntegrationTests project, I initially tried to update Elastic.Elasticsearch.Xunit and Elastic.Elasticsearch.Managed, but left the tests as they were, as I wasn't able to make the tests working at all. They are also breaking on GitHub (at least in my fork), due to missing HTTP resources. This is completely out of my depth, so I would not spend more time tinkering with this project

[mivano]

Thanks @nenadvicentic for your efforts! Great to see some improvements and needed updates. Unfortunately the tests are tricky. I will try to find some time to see if I can get them to work. Furthermore, we need to make sure we document the changes, preferably in the changes.md and readme.md file.

[nenadvicentic]

@mivano I updated both ReadMe.md and Changes.md to reflect the changes.

Also, I investigated tests issue a bit deeper, including updating GitHubActionsTestLogger NuGet package, which gave a bit clearer messages.

Firstly, integration tests are not even run on GitHub CI. Unit tests are the once throwing error.

I think updating .github/workflows/cicd.yaml file would fix current issue (but even if it does not fix the issues, we should update SDK):

      - name: Install .NET SDK
        uses: actions/[email protected]
        with:
-          dotnet-version: '3.1.x'
+          dotnet-version: '7.0.x'

However, I do not have permissions to do that.

My suggestion to you is that we merge this PR sooner, rather than later to the dev branch, since it contains quite a lot of code changes and cleaning up. We cannot make this Sink perfect with a single PR and we should keep this PR on the topic (solving problem of TypeName). Once this is merged, we make further improvements from there, and experiment with feature branches (e.g. for Elastic Common Schema topic).

[mivano]

Totally agree, let's merge it to dev (which will create pre-release packages anyway) and iterate from there.

[mivano]

I updated the dotnet-version as well and changed the test to only run for net6 and net7. This caused the tests to complete. There are now new ci packages in the GitHub packages.