DRAFT: Reduce duplicate peer traffic for ledger data

[ximinez]

Recreates the changes from #5126 (reverted in #5300), and the related bug fix once I find it.

• Drop duplicate outgoing TMGetLedger messages per peer
  • Allow a retry after 30s in case of peer or network congestion.
  • Addresses RIPD-1870
  • (Changes levelization. That is not desirable, and will need to be fixed.)
• Drop duplicate incoming TMGetLedger messages per peer
  • Allow a retry after 15s in case of peer or network congestion.
  • The requestCookie is ignored when computing the hash, thus increasing the chances of detecting duplicate messages.
  • With duplicate messages, keep track of the different requestCookies (or lack of cookie). When work is finally done for a given request, send the response to all the peers that are waiting on the request, sending one message per peer, including all the cookies and a "directResponse" flag indicating the data is intended for the sender, too.
  • Addresses RIPD-1871
• Drop duplicate incoming TMLedgerData messages
  • Addresses RIPD-1869
• Improve logging related to ledger acquisition
• Class "CanProcess" to keep track of processing of distinct items

---

High Level Overview of Change

Context of Change

Type of Change

• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] Refactor (non-breaking change that only restructures code)
• [ ] Performance (increase or change in throughput and/or latency)
• [ ] Tests (you added tests for code that already exists, or your new feature included in this PR)
• [ ] Documentation update
• [ ] Chore (no impact to binary, e.g. .gitignore, formatting, dropping support for older tooling)
• [ ] Release

API Impact

• [ ] Public API: New feature (new methods and/or new fields)
• [ ] Public API: Breaking change (in general, breaking changes should only impact the next api_version)
• [ ] libxrpl change (any change that may affect libxrpl or dependents of libxrpl)
• [ ] Peer protocol change (must be backward compatible or bump the peer protocol version)

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 24.34988% with 320 lines in your changes missing coverage. Please review.

Project coverage is 77.9%. Comparing base (1c99ea2) to head (3a685d6).

Files with missing lines	Patch %	Lines
src/xrpld/overlay/detail/PeerImp.cpp	0.0%	210 Missing :warning:
src/xrpld/overlay/detail/ProtocolMessage.h	0.0%	32 Missing :warning:
src/xrpld/app/misc/NetworkOPs.cpp	21.4%	22 Missing :warning:
src/xrpld/overlay/detail/PeerSet.cpp	20.8%	19 Missing :warning:
src/xrpld/app/ledger/detail/InboundLedgers.cpp	67.9%	17 Missing :warning:
src/xrpld/app/ledger/detail/InboundLedger.cpp	54.5%	5 Missing :warning:
src/xrpld/app/ledger/InboundLedger.h	60.0%	4 Missing :warning:
src/xrpld/app/misc/HashRouter.cpp	60.0%	4 Missing :warning:
src/xrpld/app/ledger/detail/LedgerMaster.cpp	0.0%	3 Missing :warning:
src/xrpld/app/misc/HashRouter.h	77.8%	2 Missing :warning:
... and 2 more

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #5301     +/-   ##
=========================================
- Coverage     78.1%   77.9%   -0.2%     
=========================================
  Files          795     796      +1     
  Lines        68598   68921    +323     
  Branches      8281    8441    +160     
=========================================
+ Hits         53584   53668     +84     
- Misses       15014   15253    +239     

Files with missing lines	Coverage Δ
include/xrpl/basics/base_uint.h	96.9% <100.0%> (+<0.1%)	:arrow_up:
include/xrpl/protocol/LedgerHeader.h	100.0% <ø> (ø)
src/xrpld/app/ledger/detail/TimeoutCounter.cpp	88.4% <100.0%> (+1.3%)	:arrow_up:
src/xrpld/app/ledger/detail/TimeoutCounter.h	100.0% <ø> (ø)
src/xrpld/app/misc/NetworkOPs.h	100.0% <ø> (ø)
src/xrpld/overlay/Peer.h	100.0% <ø> (ø)
src/xrpld/overlay/detail/PeerImp.h	12.8% <ø> (ø)
src/xrpld/overlay/detail/ProtocolVersion.cpp	86.4% <ø> (ø)
include/xrpl/basics/CanProcess.h	95.5% <95.5%> (ø)
src/xrpld/app/consensus/RCLConsensus.cpp	62.2% <0.0%> (ø)
... and 10 more

... and 5 files with indirect coverage changes

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.