tsaikd
Splitting one input into multiple outputs
Hello, Basically, I have one input record which is JSON of array of 1000 records, I need to split them into individual 1000 records and run filters on them and insert them individually into ELK. How can I do it?
Hi, I'm interested by the same feature. Based on my understanding of the code unfortunately it looks like Gogstash filters only allow exactly 1 (one) event out (as in one event in -> one event out).
Can anyone confirm this ?
I have several use-cases where it'd be useful for filters to allow either zero, one, or multiple events out:
- "splitting" as requested by Mayur above is one of those where we would let multiple events out of the filter.
- "dropping" events would also be a use-case for a filter that returns zero events (although this can already be done through conditional outputs I feel it might be interesting to remove unnecessary processing of events we want to discard).
- Lastly, I'd like to implement event-pairing filters, similar to Logstash "ellapsed" and "aggregate" filters. Those filters hold events in a buffer before releasing the combined result, therefore such filters might need to return zero event first, then two at once.
At the beginning, it's designed to process logs one by one. If you want to handle M inputs and N outputs, M and N are unsigned integers, some interfaces need to be changed.
If you need something to split your input have a look at Gollum, also written in go.
Gollum is an n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations.
It will not solve all of your issues but see if you can use it for something.
