Elasticsearch.Extensions.Logging Just Log ECS format to File from .NET [FEATURE]

[ghost]

I really like Elasticsearch.Extensions.Logging. It is a piece of cake to implement. However, not all of my apps are logging to Elastic. Some are logging to Splunk. ECS is a superior schema. I would like to utilize it as the common logging output. Splunk should have no problem with it.

How can I just log from .NET logging to a file with ECS format? Can you provide me some guidance here? Most are using NLog.

[snakefoot]

There is an example here for using EcsLayout with NLog Console Target:

https://github.com/elastic/ecs-dotnet/blob/master/src/Elastic.CommonSchema.NLog/README.md

Should be trivial to also use it with NLog FileTarget (Change from type="Console" to type="File")

[ghost]

I knew about NLog but was hoping for without to minimize dependencies. However, NLog does provide a rich set of features. I will check it out. Thank You!

[sgryphon]

This would be a good enhancement, although currently I'm focusing on getting Extensions to GA, so that it is broadly available and easy to add.

Currently the logger submits the log event to the elastic client, which takes care of sending it to the server, etc, although bypassing that to write the output to a file won't be too much of a change.

Some considerations though:

• Should it be an option on this logger, although it is only using the ECS format and not actually logging to Elasticsearch, or a separate logger provider package?
• Assuming you mean the JSON format that would be submitted, what formatting options would you want, e.g. whitespace, newlines, indenting, etc, or just single line output?
• Structure of the file; each JSON fragment would make sense, but would you be expecting a starting array "[" and then commas "," in between, or just fragments separate by newlines (e.g. JSON Lines format, https://jsonlines.org/)
• What file management features would you want, e.g. timestamped files, maximum size, rolling files, cleanup, etc. (for one such approach, see https://github.com/sgryphon/essential-logging/tree/master/src/Essential.LoggerProvider.RollingFile)

[sgryphon]

Thinking about these, and based on what you want (import into Splunk, or some other tool), logging using ECS JSON Lines format, with rolling file options, seems to fit what you need.

Either a new logger EcsJsonLinesLogger or some sort of flag like EcsJsonLinesFormat, although a new logger may be better otherwise the configuration and code becomes a big mess with two parts (file or Elasticsearch) only one of which is used at a time.

Extracting out the common gathering parts, and composing them into two different loggers, may be neater.

[ghost]

This is great work. I will take a look when I get a chance. Just replicating the NLog rolling file features will make it great. It would just have to be something that FileBeat and Splunk would work with. Splunk will work with most valid JSON whether it is an array or just blobs separated by CRLF. I have used FileBeat with NLog JSON delimited with CRLF.

I was looking at doing something like it. However, I would probably just end up using NLog since I am usually pressed for time. The NLog feature set is rich along with the community. ECS negates most of the formatting feature set. The targets are nice. There is probably already an elastic target.

I was also looking to experiment with an Eventsource to ECS mapping too. Eventsource is the native .NET instrumentation. It really is the holy grail of logging/instrumentation since it is captured in Perfview with all of the other .NET eventsources. However, developers don't seem to like coding an eventsource.

[Mpdreamz]

I personally feel if you want to log to file you should use NLog or Serilog or @sgryphon's Essential.LoggerProvider.RollingFile.

There are tons of edgecases to logging to file that are already better handled by existing logging frameworks. Probably one of the major reasons there is no FileLoggerProvider OOTB from Microsoft either. If there was one we'd definitely look into supporting that.

Elastic.CommonSchema.NLog helps you to log to file in ECS format with NLog. Elastic.CommonSchema.Serilog ships with EcsTextFormatter that allows you to log to any sink accepting an ITextFormatter (e.g console/file) in ECS format.

It might make sense to ship a custom Console Logger Formatter though.

This would make it super easy to log ECS logs to stdout which is useful in docker/k8s setups.