protodetect: improve DCERPC UDP probing parser and simplify app-layer-detect-proto.c code v2

[ilya-bakhtin]

Make sure these boxes are signed before submitting your Pull Request -- thank you.

• [x] I have read the contributing guide lines at https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html
• [x] I have signed the Open Information Security Foundation contribution agreement at https://suricata.io/about/contribution-agreement/ (note: this is only required once)
• [ ] I have updated the user guide (in doc/userguide/) to reflect the changes made (if applicable)
• [x] I have created a ticket at https://redmine.openinfosecfoundation.org/projects/suricata/issues (if applicable)

Link to ticket: https://redmine.openinfosecfoundation.org/issues/7111

Describe changes: This is a replacement of https://github.com/OISF/suricata/pull/11541 There are 2 commits. The first one is intended to improve DCERPC UDP detection. False positives resulted in improper work of the detection framework. The second commit simplifies the detection framework function AppLayerProtoDetectGetProto. It previously contained a bug that combined with a false positive in DCERPC resulted in incorrect reporting of DNS flow direction.

Provide values to any of the below to override the defaults.

• To use an LibHTP, Suricata-Verify or Suricata-Update pull request, link to the pull request in the respective _BRANCH variable.
• Leave unused overrides blank or remove.

SV_REPO= SV_BRANCH=https://github.com/OISF/suricata-verify/pull/2008 SU_REPO= SU_BRANCH= LIBHTP_REPO= LIBHTP_BRANCH=

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 82.61%. Comparing base (304271e) to head (fdd0a27).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11644      +/-   ##
==========================================
- Coverage   82.61%   82.61%   -0.01%     
==========================================
  Files         919      919              
  Lines      248997   248992       -5     
==========================================
- Hits       205717   205712       -5     
  Misses      43280    43280              

Flag	Coverage Δ
fuzzcorpus	60.88% <100.00%> (-0.01%)	:arrow_down:
livemode	18.66% <0.00%> (+<0.01%)	:arrow_up:
pcap	43.97% <100.00%> (-0.17%)	:arrow_down:
suricata-verify	61.87% <100.00%> (-0.03%)	:arrow_down:
unittests	59.01% <50.00%> (+<0.01%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

[ilya-bakhtin]

A question about > Commit messages : 🟠 Could you add the ticket reference in the messages ?

Is it necessary to prepare the new branch or 'commit --amend' and forced push is suitable?

[catenacyber]

We prefer a new branch and PR indeed

[suricata-qa]

Information: QA ran without warnings.

Pipeline 22276

[ilya-bakhtin]

new PR https://github.com/OISF/suricata/pull/11679

[victorjulien]

replaced by #11679