dcerpc incomplete/v4

[inashivb]

Link to redmine ticket: https://redmine.openinfosecfoundation.org/issues/5699

Previous PR: #9270

Changes since v3:

• fix pre existing issue: error out on invalid header, use diff retvals (found by fuzzer post incomplete API changes)

[codecov[bot]]

Codecov Report

Merging #9276 (2f73811) into master (9a33c53) will decrease coverage by 0.01%. The diff coverage is 94.11%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9276      +/-   ##
==========================================
- Coverage   82.42%   82.41%   -0.01%     
==========================================
  Files         968      968              
  Lines      273952   273714     -238     
==========================================
- Hits       225807   225594     -213     
+ Misses      48145    48120      -25     

Flag	Coverage Δ
fuzzcorpus	64.71% <94.11%> (+0.02%)	:arrow_up:
suricata-verify	60.79% <82.35%> (-0.02%)	:arrow_down:
unittests	62.90% <82.35%> (-0.04%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

[suricata-qa]

WARNING:

field	baseline	test	%
	SURI_TLPR1_stats_chk
.app_layer.flow.dcerpc_tcp	4188	43	1.03%
.app_layer.tx.dcerpc_tcp	5949	4165	70.01%
.app_layer.error.dcerpc_tcp.parser	6248	10393	166.34%

Pipeline 15350

[inashivb]

This one is blocked bc:

• I need to make a minimal reproducer of the pcap that causes the massive stats deviations and,
• confirm/deny whether this PR is correct Will update once I have something working on that front.

[catenacyber]

Did you get a minimal pcap reproducer for stats deviation ? Can I help somehow here ?

[inashivb]

Did you get a minimal pcap reproducer for stats deviation ?

No. I couldn't pull out the particular full dcerpc stream(s) that caused this. But, I haven't checked in a while now.

Can I help somehow here ?

Yes, if you could tell how to go about extracting such streams from very large pcaps? The one I'm concerned with for the stats deviation in this PR is 190GB.

Please note this is what I had tried: Filter out all the dcerpc traffic with tshark and run Suricata with midstream enabled on it. But, my extracted pkts were broken so I did something wrong. Victor said we shouldn't hack it and extract full streams. I was unable to figure out the correct filter for that at the time.

Challenges I had faced:

• tshark tries to load entire pcap at once so memory was an issue at first
• Saving the pkts extracted w filter requires more than just saving the extracted pkts

[catenacyber]

Can I help somehow here ?

Yes, if you could tell how to go about extracting such streams from very large pcaps? The one I'm concerned with for the stats deviation in this PR is 190GB.

Please note this is what I had tried: Filter out all the dcerpc traffic with tshark and run Suricata with midstream enabled on it. But, my extracted pkts were broken so I did something wrong. Victor said we shouldn't hack it and extract full streams. I was unable to figure out the correct filter for that at the time.

I agree with Victor. Can you have some filter based on ports ?

I also added some comments inline. The bytes_consumed field seems to need more changes

[ct0br0]

Extracting a pcap with ports 135, 139 and 445 from TLPR1 is around 415MB and has 4140 .app_layer.error.dcerpc_tcp.parser but splitting by IP only has 13 from 192.168.0.1 which isn't what I'd expect. I will see if I can find anything that may help more.

[inashivb]

Extracting a pcap with ports 135, 139 and 445 from TLPR1 is around 415MB and has 4140 .app_layer.error.dcerpc_tcp.parser but splitting by IP only has 13 from 192.168.0.1 which isn't what I'd expect. I will see if I can find anything that may help more.

Thanks a lot! Actually, I'm also interested in the 415MB pcap that has 4k+ dcerpc errors.

[suricata-qa]

WARNING:

field	baseline	test	%
	SURI_TLPR1_stats_chk
.app_layer.flow.dcerpc_tcp	4188	43	1.03%
.app_layer.tx.dcerpc_tcp	5949	4165	70.01%
.app_layer.error.dcerpc_tcp.parser	6248	10393	166.34%

Pipeline 15350

[catenacyber]

Closing this PR as stale, let me know if I can help here Shivani