Batch filtering invalid transactions before forwarding

[tao-stones]

Problem

Filtering invalid packets (eg too old, or already processed) at end-of-slot was accidentally removed. It is worth to add it back to avoid forwarding unnecessary (eg, invalid) packets.

When checking if packet is invalid, should keep in mind not to lock bank's blockhash_queue and status_cache for every transactions, as that's be bad for overall performance. This requires to change the forwarder's workflow from transaction-by-transaction to batch-by-batch.

Summary of Changes

• Move packets filtering to the beginning of forwarding process, so the logic only responsible for forwarding. (leader still filters invalid packets at consume_buffered_packets as usual).
• Sanitizing batch of packets into transactions, validating transactions with bank in batch to avoid excessive locking on bank's blockhash and status cache.
• Refactor ForwardPacketBatchesByAccounts to re-sued sanitized transactions from above step to avoid extra packet sanitizing;
• Report metrics

Fixes #26414

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[tao-stones]

Wanted to add the filtering invalid packets to forwarder. It has been awhile, started to wondering if this is still relevant, even tho I still think filter out invalid packets from unprocessed buffer before forwarding is a valid step. Can you take a quick look to see if this is necessary, I'll update the tests and bench soon. @carllin @AshwinSekar

[tao-stones]

@carllin @AshwinSekar this should help banking queue throughput, can it be reviewed when you have chance?