Add timeout for BanksServer.process_transaction methods

[kevinheavey]

Problem

All the process_transaction* methods in impl Banks for BanksServer will spend 60 seconds trying to find a signature before tarpc gives up with a DeadlineExceeded error. 60 seconds is a long time and (correct me if I'm wrong), it should take a very short length of time for BanksServer to be certain that the transaction signature won't be found - perhaps one second or less? After all BanksServer is only used for local testing, right?

The 60-second hang is encountered whenever a transaction is sent but not committed (see #30527), so if possible it would be better to catch cases where the transaction isn't committed and report the error back to the user, but I don't have an opinion on how feasible that is. Another option might be automatically retrying the transaction - that would work for #30527 at least.

Proposed Solution

Change the value of the last_valid_block_height that gets calculated in BanksServer.process_transaction_with_commitment_and_context so that it's only a few blocks ahead of the current block (currently it waits until the blockhash is invalid which is longer than 60 seconds anyway). Then poll_signature_status will return after a reasonable length of time.

The number of blocks to wait could be configurable if people want that. If we make it configurable then the configuration should bubble up all the way ProgramTest struct since that's how most people instantiate BanksClient and BanksServer.

As mentioned above, there might be a better way to fix this, but I think we can agree that the 60-second hangs in program-test workflows are bad.

[kevinheavey]

Ok the last_valid_block_height solution doesn't seem to work. Looks like the block_height doesn't get incremented so you're never gonna reach any last_valid_block_height that's greater than the current block_height