flow: compute stat counters again

[catenacyber]

Link to redmine ticket: https://redmine.openinfosecfoundation.org/issues/5317

Describe changes:

• flow: compute stat counters again as broken by commit b3599507f4eb891841417575587d690ea13fe6c0

Variables like FlowTimeoutCounters.clo was always 0 and never changed. It is sad that no compiler/static analysis tool caught this...

There is no S-V tests about flow timeouts (because of reading pcaps, flow manager thread does not get the time to do its job)... Are there some in QA ?

[codecov[bot]]

Codecov Report

Merging #7358 (58d7223) into master (323fe1c) will decrease coverage by 0.04%. The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #7358      +/-   ##
==========================================
- Coverage   75.84%   75.80%   -0.05%     
==========================================
  Files         656      656              
  Lines      190093   190104      +11     
==========================================
- Hits       144172   144101      -71     
- Misses      45921    46003      +82     

Flag	Coverage Δ
fuzzcorpus	60.43% <0.00%> (-0.05%)	:arrow_down:
suricata-verify	51.69% <0.00%> (-0.06%)	:arrow_down:
unittests	61.02% <0.00%> (-0.01%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

[suricata-qa]

ERROR:

ERROR: QA failed on tlpw1_files_sha256.

field	test	baseline	%
	tlpr1_stats_chk
.app_layer.error.ftp-data.parser	0	395	0.0%

Pipeline 7354

[suricata-qa]

ERROR:

ERROR: QA failed on tlpw1_files_sha256.

field	test	baseline	%
	tlpr1_stats_chk
.app_layer.error.ftp-data.parser	0	395	0.0%

Pipeline 7354 WARNING: THERE IS A KNOWN BAD BASELINE WITH PACKET DROPS. bE MINDFUL OF ANY RESULTS.

[victorjulien]

In master we have a new set of counters that I think is more useful

flow.end.state.new                            | Total                     | 35013
flow.end.state.established                    | Total                     | 5342
flow.end.state.closed                         | Total                     | 109812
flow.end.tcp_state.syn_sent                   | Total                     | 1358
flow.end.tcp_state.established                | Total                     | 296
flow.end.tcp_state.fin_wait1                  | Total                     | 32
flow.end.tcp_state.fin_wait2                  | Total                     | 42
flow.end.tcp_state.time_wait                  | Total                     | 5
flow.end.tcp_state.close_wait                 | Total                     | 9
flow.end.tcp_state.closed                     | Total                     | 109807

So lets fully remove these old ones instead of fixing them.

[catenacyber]

So flow.mgr.new_pruned is now in flow.end.state.new, right ?

[victorjulien]

So flow.mgr.new_pruned is now in flow.end.state.new, right ?

It's not exactly the same thing, but ya. Flows can be pruned by worker threads too.

[catenacyber]

So flow.mgr.new_pruned is now in flow.end.state.new, right ?

It's not exactly the same thing, but ya. Flows can be pruned by worker threads too.

Is it really about timed out flows ?

cf https://github.com/OISF/suricata/pull/7790

[victorjulien]

So flow.mgr.new_pruned is now in flow.end.state.new, right ?

It's not exactly the same thing, but ya. Flows can be pruned by worker threads too.

Is it really about timed out flows ?

cf #7790

Both are about flows that are evicted from the hash. Can be multiple reasons. Most common in timeout, but in resource stress cases we forcefully evict sooner.