output/eve: reduce fflush call count

[jlucovsky]

Continuation of #9802

Link to redmine ticket: 3449

Describe changes:

• Add EVE configuration parameter to control buffering: buffer-size. When 0, unbuffered I/O is used; other values are used to set the stdio buffer size. The value is outputs.eve-log.buffer-size
• Add Suricata configuration parameter -- flush-interval -- to set cadence for Suricata periodically directing detect threads to flush EVE output. To be used in conjunction with buffer-size. Set logging.flush-interval to the number of seconds Suricata should periodically cause the EVE output to be flushed. The default value is 0 which instructs Suricata to never cause the EVE output to be flushed.
• Add mechanism to periodically send pseudo packets to detect threads as a way to trigger flush. Controlled by flush-interval
• Add "log flusher" thread when flushing is configured (logging.flush-interval is between 1 and 69).

Updates:

• Added file missing from git
• Modified log-flush thread to sleep in a series of 500ms increments to respond better to shutdown scenarios
• Checked detection context pointers that could possibly become state in flush and packet injection functions.

hyperfine results

Suricata configurations

Threaded output off

• suricata.yaml (default values)
• suricata-buffer-flush.yaml (flush every 2 seconds with 16kb buffer)
• suricata-buffering.yaml (16kb buffer)

Threaded output

• suricata.yaml
• suricata-buffer-flush.yaml
• suricata-buffering.yaml

Benchmark 1: ./src/suricata -c suricata.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll
  Time (mean ± σ):     411.279 s ±  2.769 s    [User: 1375.083 s, System: 135.731 s]
  Range (min … max):   409.550 s … 414.473 s    3 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (414.473 s). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Benchmark 2: ./src/suricata -c suricata-buffer-flush.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll
  Time (mean ± σ):     413.066 s ±  6.341 s    [User: 1377.730 s, System: 122.352 s]
  Range (min … max):   406.132 s … 418.569 s    3 runs

Benchmark 3: ./src/suricata -c suricata-buffering.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll
  Time (mean ± σ):     407.890 s ±  4.460 s    [User: 1370.222 s, System: 121.522 s]
  Range (min … max):   403.176 s … 412.042 s    3 runs

Benchmark 4: ./src/suricata -c suricata-threaded.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll
  Time (mean ± σ):     397.753 s ± 28.644 s    [User: 1363.991 s, System: 132.469 s]
  Range (min … max):   366.626 s … 423.003 s    3 runs

Benchmark 5: ./src/suricata -c suricata-buffer-flush-threaded.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll
  Time (mean ± σ):     414.763 s ± 12.046 s    [User: 1380.227 s, System: 121.760 s]
  Range (min … max):   401.456 s … 424.925 s    3 runs

Benchmark 6: ./src/suricata -c suricata-buffering-threaded.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll
  Time (mean ± σ):     418.456 s ±  2.223 s    [User: 1379.530 s, System: 122.226 s]
  Range (min … max):   417.039 s … 421.019 s    3 runs

Summary
  './src/suricata -c suricata-threaded.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll' ran
    1.03 ± 0.07 times faster than './src/suricata -c suricata-buffering.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll'
    1.03 ± 0.07 times faster than './src/suricata -c suricata.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll'
    1.04 ± 0.08 times faster than './src/suricata -c suricata-buffer-flush.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll'
    1.04 ± 0.08 times faster than './src/suricata -c suricata-buffer-flush-threaded.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll'
    1.05 ± 0.08 times faster than './src/suricata -c suricata-buffering-threaded.yaml -S ~/suricata.rules -r ~/pcap -l /tmp/ll'

Provide values to any of the below to override the defaults.

To use a pull request use a branch name like pr/N where N is the pull request number.

Alternatively, SV_BRANCH may also be a link to an OISF/suricata-verify pull-request.

SV_REPO=
SV_BRANCH=
SU_REPO=
SU_BRANCH=
LIBHTP_REPO=
LIBHTP_BRANCH=

[codecov[bot]]

Codecov Report

Attention: 174 lines in your changes are missing coverage. Please review.

Comparison is base (d2b25af) 82.42% compared to head (b794858) 82.38%. Report is 155 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9832      +/-   ##
==========================================
- Coverage   82.42%   82.38%   -0.04%     
==========================================
  Files         972      973       +1     
  Lines      273929   274215     +286     
==========================================
+ Hits       225788   225922     +134     
- Misses      48141    48293     +152     

Flag	Coverage Δ
fuzzcorpus	64.27% <46.68%> (-0.03%)	:arrow_down:
suricata-verify	61.05% <51.93%> (-0.04%)	:arrow_down:
unittests	62.84% <40.33%> (-0.04%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

[suricata-qa]

Information: QA ran without warnings.

Pipeline 16634

[suricata-qa]

Information: QA ran without warnings.

Pipeline 16634

[jlucovsky]

Continued in #11059