builder: Consistency issue on Arbitrum with conditional sender

[dancoombs]

Describe the bug When using the conditional sender we use 1 RPC endpoint in the builder server, and a different endpoint in the transaction sender (Arbitrum's endpoint). This can lead to consistency issues if the server RPC gets notified of a nonce change prior to the conditional RPC processing the block change. The builder server uses the transaction sender to get the transaction status after the nonce change.

This leads the builder to believe that an external transaction was mined, and not the one its looking for the status for.

To solve this, we should ensure that the same RPC that is being used to track nonces is also being used to track transaction status. We should likely start viewing the conditional endpoints as purely submission endpoints and use the regular RPC for status tracking. We won't get pending transaction support this way, and will need to implement #295 first.

To reproduce Send bundle transactions on Arbitrum Goerli. Reproduces about 50% of the time.

Expected behavior The builder should have a consistent view of the chain. If its nonce increases, it should be able to see the status of the transaction that was just mined.

Logs (added some extra logs in a debug build)

  2023-08-13T19:08:10.017849Z  INFO rundler::builder::transaction_tracker: Nonce changed from 132 to 133. Pending transactions: [PendingTransaction { tx_hash: 0x465a215b1ef82c28aefebd4935a3693163766d2585e939212b9bcbb0c5cfd237, gas_fees: GasFees { max_fee_per_gas: 100000000, max_priority_fee_per_gas: 0 }, attempt_number: 0 }]
    at src/builder/transaction_tracker.rs:261

  2023-08-13T19:08:10.141960Z  INFO rundler::builder::sender::conditional: tx_hash 0x465a215b1ef82c28aefebd4935a3693163766d2585e939212b9bcbb0c5cfd237 tx: None
    at src/builder/sender/conditional.rs:61

  2023-08-13T19:08:10.142100Z  INFO rundler::builder::server: Nonce used by external transaction
    at src/builder/server.rs:212

  2023-08-13T19:08:10.142281Z  INFO rundler::common::emit: Transaction failed because nonce was used by another transaction outside of this Rundler.    Nonce: 132    Entrypoint: 0x5ff137d4b0fdcd49dca30c7cf57e578a026d2789

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.