gogstash

gogstash copied to clipboard

I'm looking to process JSON logs with fields like this:

{  "http.method": "GET" }

...and many more similar fields with . in the name.

I can use the json filter to transform this into fields, but none of the other filters are able to manipulate these fields. For example with these filters:

filter:
  - type: json
  - type: remove_field
    fields: ["http.method"]

...running the above JSON through yields:

{
	"@timestamp": "2020-08-21T02:26:06.158181Z",
	"host": "MyHost",
	"http.method": "GET",
	"message": "{\"http.method\":\"GET\"}"
}

There's no way to remove the field! Other filters have a similar issue.

The problem is that getPathValue always interprets its input as a path expression and not necessarily a literal key into the map. Reference: https://github.com/tsaikd/gogstash/blob/d4613914309c6655ff2234eb15a4fb013c0d2ee1/config/logevent/pathvalue.go#L87

Logstash has field reference syntax to handle these different cases. I can understand that gogstash may not want such complexity, but it does feel like there should be a solution to my problem. Otherwise I think gogstash will be unusable for me.

How about adding an extra syntax (e.g. single quote) to handle this case? Or any other suggestions?

filter:
  - type: remove_field
    fields: ["'http.method'"]

How about adding an extra syntax (e.g. single quote) to handle this case? Or any other suggestions?

filter:
  - type: remove_field
    fields: ["'http.method'"]

How about this way:

"http\\.method" to escape dots?

Or the logstash like way: "[http.method]"?