Release new version of ECS.NET 8.4

[Mpdreamz]

• [ ] sanity check the process field set (@gregkalapos)
• [x] Extend the code generator to include richer meta information in xml documentation. (@Mpdreamz )
• [x] Merge https://github.com/elastic/ecs-dotnet/pull/110 as it has a lot of integration tests using the new types. (@gregkalapos)
• [ ] We will aim for an 8.4.0-alpha first to validate our build tooling. (@mpdreamz)
• [ ] Breaking changes documentation. Include an paragraph describing what's expected to break with this new major version and why. (@mpdreamz)

[bonyaroslav]

@Mpdreamz @gregkalapos Any news when we can use it? We're waiting for many months already

[odin568]

Same for me. Eager to use updated version to get rid of some workarounds

[lewinskimaciej]

Is this package still being maintained? We desperately need https://github.com/elastic/ecs-dotnet/issues/167 but it seems that there hasn't been a release in 1,5 years. Is there something we can do to help push this out?

[Mpdreamz]

Still maintained folks! Thanks for the nudges for an update.

I have been on leave for an extended period of time but am back working on getting an update out the door. Any feedback on the upcoming pre-release would be most welcomed!

Will comment on this issue when it goes up on nuget.

[Mpdreamz]

Heads up here that we just shipped version 8.4.0-alpha1 to nuget.

I intend to wait two weeks to collect feedback on this new version before shipping it as a GA release (8.4.0).

A LOT of changes have gone in since we last release 1.6.0-alpha1, you can see the full list of changes here

Here are some highlights:

New integrations

• Elasticsearch.Extensions.Logging an integration with Microsoft.Extensions.Logging that writes logs directly to Elasticsearch or Elastic Cloud (thank you @sgryphon for all the help)
• Elastic.CommonSchema.Serilog.Sink a Serilog sink that writes ecs logs directly to Elasticsearch or Elastic Cloud
• Elastic.CommonSchema.Log4Net Log4Net integration that logs to files in ecs-logging format. (thank you @andreycha!)
• Elastic.Ingest.Elasticsearch.CommonSchema a specialization of Elastic.Ingest.Elasticsearch to push data to datastreams indices that is fully ECS aware and can bootstrap the target datastreams/indices confirming ECS mappings.

Big Updates:

• Completely rewrote the code generator from scratch to accomodate the ECS spec format changes since 1.6 to 8.x.
• Spun off the ingest components into their own seperate libraries: elastic/elastic-ingest-dotnet
• Elastic.CommonSchema was completely rearchitected as we rewritten the code generator from scratch to align with the ECS specification changes.
  • EcsDocument now has an AssignField(path, value) method that can dynamically assign ECS fields based on path. This allows all the structured logging integrations to directly assign ECS fields too.
  • Through CreateNewWithDefaults all logging integrations share a common base default for how ecs documents get created and enriched.
• NLog integration has seen continued improvements from @snakefoot (Can't thank you enough for all your efforts!)
• Moved CI back onto Github Actions (thank you @v1v).

Breaking Changes.

• The root object for ECS documents in Elastic.CommonSchema was renamed from Base to EcsDocument.

[HubDevUser]

Can you please include .netstandard2.0 in the official 8.4.0 release. We need to consume this in fullframework, the alpha1 release only contains .netstandard2.1 assemblies @Mpdreamz

[reydelleon-skuvault]

@Mpdreamz Are there plans to update documentation to show how to put all these together? For example, how to enhance the EcsDocument with custom fields.

I noticed that you're putting forward an alternative to Serilog.Sinks.Elasticsearch. it would be helpful to see some examples of how you would configure your sink to achieve a similar experience (knowing that they don't have the same features) than that of the contrib sink. Many of us have been using that other sink for a long time and would like to make any possible migration as painless as possible.

[Mpdreamz]

@reydelleon-skuvault absolutely, docs are a big prerequisite for the 8.4.0 release.

I tried to capture some of the main differences between the sinks here for now: https://github.com/elastic/ecs-dotnet/tree/main/src/Elastic.CommonSchema.Serilog.Sink#comparison-with-serilogsinkselasticsearch

Any feedback or additional questions you may have are more than welcomed!

[Mpdreamz]

A quick extremely raw example of custom EcsDocument implementation is here: https://github.com/elastic/ecs-dotnet/blob/main/src/Elastic.CommonSchema.BenchmarkDotNetExporter/Domain/BenchmarkDocument.cs#L16

[HubDevUser]

How can i set a custom index, it always defaults to ecs-dotnet-logs ?

[Mpdreamz]

How can i set a custom index, it always defaults to ecs-dotnet-logs ?

Using what integration?

[HubDevUser]

Using aspnet core similar to the example you added earlier examples/aspnetcore-with-extensions-logging

[Mpdreamz]

Using aspnet core similar to the example you added earlier examples/aspnetcore-with-extensions-logging

If you are using Elasticsearch.Extensions.Logging you can configure it in appsettings.json

{
  "Logging": {
    "Elasticsearch": {
      "ShipTo": {
        "NodePoolType": "Static",
        "NodeUris": [ "http://localhost:9200" ]
      },
      "DataStream": {
        "DataSet": "my.application"
      }
    },
    "LogLevel" : {
      "Default": "Trace",
      "Microsoft": "Warning"
    }
  }
}

This will log to logs-my.application-default.

You can also configure this in the application builder.

builder.Host.ConfigureLogging((_, loggingBuilder) =>
{
	loggingBuilder.AddElasticsearch(opts =>
	{
		opts.DataStream = new DataStreamNameOptions
		{
			Type = "logs", DataSet = "dotnet", Namespace = "default"
		}
	}),

[Mpdreamz]

Can you please include .netstandard2.0 in the official 8.4.0 release. We need to consume this in fullframework, the alpha1 release only contains .netstandard2.1 assemblies @Mpdreamz

I've just pushed 8.4.0-alpha2 with netstandard2.0 support.

[HubDevUser]

It doesn't contain a .netstandard2.0 under libs, can you take a look please ?

[reydelleon-skuvault]

@Mpdreamz I'm trying to use the new sink but I'm not seeing a way to specify auth credentials when configuring it. There is nothing about this in the README and looking through the code I wasn't able to identify a way to do this.

Can you provide some guidance on this and perhaps add something in the examples in the README?

[lewinskimaciej]

@Mpdreamz I too can't seem to find a way of setting auth on new Serilog Elastic sink. I think this would be achievable by exposing setter for ElasticsearchSinkOptions.Transport or some other, maybe nicer way. Currently it's always DefaultHttpTransport with default TransportConfiguration.

[HubDevUser]

@reydelleon-skuvault @lewinskimaciej see https://github.com/elastic/ecs-dotnet/pull/267#discussion_r1107859406 for example on setting up the auth options

[HubDevUser]

@Mpdreamz how can we configure for .netfullframework where ILogger is not an option?

[LiorBanai]

@HubDevUser you can use ILogger in net framework using the nuget:

		<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="7.0.0" />

[reydelleon-skuvault]

@reydelleon-skuvault @lewinskimaciej see https://github.com/elastic/ecs-dotnet/pull/267#discussion_r1107859406 for example on setting up the auth options

@HubDevUser The example you linked is for the logging provider, not the sink. @Mpdreamz We need an example that works for the sink. Do we have one?

[reydelleon-skuvault]

Found the way to do it, but it is rather convoluted if all I want to change is the authentication.

var transportConfig = new TransportConfiguration(new Uri("http://localhost:9200/"));
transportConfig.Authentication(new BasicAuthentication("elastic", "some-password"));
var transport = new DefaultHttpTransport(transportConfig);
var sinkOptions = new ElasticsearchSinkOptions(transport)
{
	...
};
loggerConfiguration
	.WriteTo.Elasticsearch(sinkOptions);

@Mpdreamz Is there a better way to do this?

[Mpdreamz]

@reydelleon-skuvault thanks for bringing this to my attention I've opened https://github.com/elastic/ecs-dotnet/pull/286 to address this usability issue.

With that you should be able to use:

.WriteTo.Elasticsearch(nodes, opts =>
{
	opts.BootstrapMethod = BootstrapMethod.Failure;
	opts.DataStream = new DataStreamName("logs", "console-example");

}, transport =>
{
	transport.Authentication();
})

[Mpdreamz]

I just pushed 8.4.0-alpha3 to nuget that includes netstandard2.0 for all projects and includes the fix to make it easier to configure the transport options when using the serilog sink.

Thanks everyone for kicking the tires!

[Mpdreamz]

I just pushed ECS.NET 8.4.0-alpha4 that includes everyones feedback and PR's much obliged to everyone kicking the tires!

I opened #291 to bump us to 8.6.0 and this includes some non breaking changes to how we bootstrap index templates. This PR will allow new version to upgrade template indices.

[reydelleon-skuvault]

@Mpdreamz Is there a good way to see logs about the sink activity? I have SelfLog enabled for Serilog and the sink configured but thought I'm not getting any data in Elasticsearch, there is no indication as to what is the cause for this. Nothing written to the self log file (is it supposed to?).

I have disabled pinging (a shot in the dark) but that didn't resolve it.

I do have logs in the Console and local file, so there is content. It is just not making it to Elasticsearch.

[Mpdreamz]

We are logging all export errors to Serilog's Self log:

https://github.com/elastic/ecs-dotnet/blob/af53e3cdb8d00858b62f6d071b442163a9bfad9f/src/Elastic.CommonSchema.Serilog.Sink/ElasticsearchSink.cs#L77

The following diagnostics method is still ugly and not intended to make it to 1.0 but we ship with something called a ChannelListener<> if you do the following.

	.WriteTo.Elasticsearch(....
    {
		ConfigureChannel = channelOpts =>  {
			channelOpts.BufferOptions = new BufferOptions { ExportMaxConcurrency = 10 };

			SomeStaticPlace = new ChannelListener<EcsDocument, BulkResponse>().Register(channelOpts);
		}
	})

What does SomeStaticPlace.ToString() report?

[reydelleon-skuvault]

@Mpdreamz This is what I get from the code you posted:

Failed publish over channel: ChannelListener.
Exported Buffers: 0
Exported Items: 0
Export Responses: 0
Export Retries: 0
Export Exhausts: 0
Inbound Buffer Read Loop Started: False
Inbound Buffer Publishes: 0
Inbound Buffer Publish Failures: 0
Outbound Buffer Read Loop Started: False
Outbound Buffer Read Loop Exited: False
Outbound Buffer Publishes: 0
Outbound Buffer Publish Failures: 0

Exception: 

That's it. No more details than that. Here is my setup (without the diagnostic code), in case I'm doing something wrong here:

loggerConfiguration
			.WriteTo.Elasticsearch<EnhancedEcsDocument>(
				new[] {new Uri("http://localhost:9200")},
				options =>
				{
					options.DataStream = new DataStreamName("logs", "myservice", environmentName.ToLowerInvariant());
					options.BootstrapMethod = BootstrapMethod.None ;
					};
				},
				transportConfig =>
					transportConfig
						.Authentication(new BasicAuthentication( "someUSer", "somePassword"))
						.DisablePing()
			);

[reydelleon-skuvault]

I ended up (successfully) shipping the logs to a different Elasticsearch server, which means two things: 1. the problem was the server I was trying to ship to and 2. The sink works as expected.

That said, I still don't know what the problem with the original server was because I didn't get anything in the Self log.

[Mpdreamz]

@reydelleon-skuvault Thanks for reporting back another instance works!

I would still love to improve the experience here and understand why the other instance does not work.

Can you run the following in a new console application? Of course using

ChannelListener<EnhancedEcsDocument, BulkResponse>? listener = null;
var waitHandle = new CountdownEvent(1);

// -- Setup Serilog --
var nodes = new[] { new Uri("http://localhost:9200") };
Log.Logger = new LoggerConfiguration()
	.MinimumLevel.Debug()
	.MinimumLevel.Override("Microsoft", LogEventLevel.Information)
	.Enrich.FromLogContext()
	.WriteTo.Elasticsearch<EnhancedEcsDocument>(nodes, opts =>
	{
		opts.BootstrapMethod = BootstrapMethod.None;
		opts.DataStream = new DataStreamName("logs", "console-example");
		opts.ConfigureChannel = channelOpts => {
			channelOpts.BufferOptions = new BufferOptions
			{
				ExportMaxConcurrency = 1,
				OutboundBufferMaxSize = 2,
				WaitHandle = waitHandle
			};
			listener = new ChannelListener<EnhancedEcsDocument, BulkResponse>().Register(channelOpts);
		};
	}, 
	transportConfig =>
		transportConfig
			.Authentication(new BasicAuthentication( "someUSer", "somePassword"))
			.DisablePing()
    )
	.CreateLogger();

// -- Log 2 items and wait for flush --
Log.Logger.Information("Writing event 1");
Log.Logger.Information("Writing event 2");

if (!waitHandle.WaitHandle.WaitOne(TimeSpan.FromSeconds(10)))
	throw new Exception($"No flush occurred in 10 seconds: {listener}", listener?.ObservedException);
else
{
	Console.WriteLine("Successfully indexed data to Elasticsearch");
	Console.WriteLine(listener);
}

Thanks in advance!