Broken host mapping for ECS

[petritz]

With Elasticsearch 7.0 elastic common schema (ECS) was introduced. This maps the hostname to host.name instead of host. Currently logstash_async fails with:

[2019-04-29T08:04:22,562][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash", :_type=>"_doc", :routing=>nil}, #LogStash::Event:0x27275849], :response=>{"index"=>{"_index"=>"logstash-2019.04.26-000001", "_type"=>"_doc", "_id"=>"tRAfaGoB1XS_Z1QomfCX", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [host] tried to parse field [host] as object, but found a concrete value"}}}}

As a workaround we changed formatter.LogstashFormatter.format to:

message = {
       	    '@timestamp': self._format_timestamp(record.created),
            '@version': '1',
       	    'host.name': self._host,  # <-- FIXED: was: 'host': ...
       	    'level': record.levelname,
            'logsource': self._logsource,
            'message': record.getMessage(),
            'pid': record.process,
            'program': self._program_name,
       	    'type': self._message_type,
        }

[eht16]

ECS sounds great, hopefully it will work and makes the logging world a bit better and more structured.

I think I will implement it here as well, just not yet sure how exactly. Probably with a new Formatter subclass to implement ECS. The previous behavior probably should stay the default to not break existing setups which are pre-ECS. So users would have to actively switch to the new scheme.

Until implemented, an easier workaround for ECS/ES7 users is to using the mutate filter plugin in Logstash to reformat the event. Actually, the mentioned host field is only one example (and it's just crashing here due to the name/type conflict) but ECS defines more fields which should be updated in the Formatter class.

[redNixon]

Despite its best intentions(and maybe a little vendor-lockin), ECS claiming ownership over generic and often utilized fieldname like "host" and "agent" I don't think was thoughtout well enough(the seemingly impossible to permanently override beats behavior to send these doesn't help matters).

If this library does adopt a ECS-by-default mode, care to not overwrite user defined fields should be taken as I do not believe the generic terms ECS is using suddenly becoming reserved words is a reasonable stance.