fluent-plugin-splunk icon indicating copy to clipboard operation
fluent-plugin-splunk copied to clipboard

Published 20 hours ago • verified publisher icon fluent

Improvement request: use_ack chunk acknowledgement, http_keepalive

Open sspreitzer opened this issue 5 years ago • 4 comments

Hi there

Thank you for this plugin!

In the current implementation of use_ack is only usable for small amounts of events. use_ack blocks the process by it's check_interval and per event POST request.

This should be implemented as per chunk POST of data and also per chunk request of acknowledgement (one POST with list of events instead of one POST per event).

Finally http_keepalive should be implemented as well.

use_ack with 10000 events is not usable.

Thank you and kind regards Sascha

sspreitzer avatar Jun 13 '19 11:06 sspreitzer

I am seeing same issue..

venky999 avatar Feb 26 '20 18:02 venky999

This should be implemented as per chunk POST of data and also per chunk request of acknowledgement (one POST with list of events instead of one POST per event).

Current implemention seems to use this approach: https://github.com/fluent/fluent-plugin-splunk/blob/master/lib/fluent/plugin/out_splunk_hec.rb#L150-L176 I'm not an expert of Splunk internal. Does this HTTP call sequence check acks for each event?

repeatedly avatar Mar 03 '20 02:03 repeatedly

More info here https://docs.splunk.com/Documentation/Splunk/8.0.2/Data/AboutHECIDXAck

venky999 avatar Mar 04 '20 10:03 venky999

Yes. My comment is based on splunk document. Could you explain why sending only 1 ackID for batch request is same as send per event, not send per chunk?

repeatedly avatar Mar 04 '20 12:03 repeatedly