logstash-filter-csv icon indicating copy to clipboard operation
logstash-filter-csv copied to clipboard

Improve performance of csv filter to be on par with dissect

Open ppf2 opened this issue 8 years ago • 1 comments

We are seeing users attempting to rewrite their csv filters to use dissect per the blog post to achieve better performance, but then running into the empty values handling issue of dissect.

A better solution is to improve the performance of the csv filter so users can just use the csv filter which is easier to setup, etc.. In fact, the blog post mentions the following:

In the near future, we will leverage some of the techniques from Dissect in the CSV filter to put its performance on a par with Dissect.

Just filing to track progress here.

ppf2 avatar Mar 22 '17 07:03 ppf2

For any interested readers the bug in the dissect filter has been fixed now. It should be usable in place of this filter but with limitations - no support for header (column names) containing events.

guyboertje avatar Jan 09 '18 10:01 guyboertje