Spurious increase of account sequence when tx fails to broadcast

[romac]

Summary of Bug

Sometimes when Hermes fails broadcasts a tx, for example because all packets it contains are redundant, it should not increment the account sequence number, otherwise it will fail to submit the next tx. In that case, the relayer then has to refresh the account sequence number before being able to successfully submit the next tx, which decreases the throughput.

See for example these logs:

LINE 1 | 2023-08-18T13:29:31.577376Z ERROR ThreadId(08) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=3e3c863e ->Destination @4-16622601; len=1}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=3e3c863e}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=4}: failed to broadcast tx with unrecoverable error response=Response { code: Err(22), data: b"", log: "packet messages are redundant", hash: Hash::Sha256(A926792B6B76076504623F9D4EE6EAD130B927EBA1FDA4974071951AC93896E2) } diagnostic=unknown TX sync response error: 22
LINE 2 | 2023-08-18T13:29:38.804661Z  WARN ThreadId(08) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=f122aceb ->Destination @4-16622602; len=1}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=f122aceb}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=5}: failed to broadcast tx because of a mismatched account sequence number, refreshing account sequence number and retrying once response=Response { code: Err(32), data: b"", log: "account sequence mismatch, expected 4, got 5: incorrect account sequence", hash: Hash::Sha256(AB357F6AD7C12727EE7EE05EE51D1ED6FE2FD17416294F2B1A232C0BFD6CFA2D) }
LINE 3 | 2023-08-18T13:29:51.441110Z ERROR ThreadId(08) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=bdb17ed6 ->Destination @4-16622604; len=1}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=bdb17ed6}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=4}: failed to broadcast tx with unrecoverable error response=Response { code: Err(22), data: b"", log: "packet messages are redundant", hash: Hash::Sha256(3890F79730A4D77CE6BA3127B27C0043D8D6A40E757A03E701C532ABFF75988E) } diagnostic=unknown TX sync response error: 22

On line 1, Hermes sends a tx with all redundant messages with account sequence 4. That tx ends up being rejected by the chain. On line 2, Hermes submits another tx with accounts sequence 5, which is rejected by the chain because the latter expects sequence 4. Hermes then refreshes the account sequence. On line 3, Hermes sends the same tx with account sequence 5, but that tx ends up being rejected because all messages are redundant, perhaps because it was too late to submit it, having had to refresh the account sequence at the previous step.

Version

master

Steps to Reproduce

TBD

Acceptance Criteria

Hermes does not increment the account sequence number when it fails to broadcast a tx.

---

For Admin Use

• [ ] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate milestone (priority) applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned

[gaia]

Also having this issue. hermes 1.8.0+39036d1: account sequence mismatch, expected 1, got 2

[romac]

@gaia Do you have any logs which exhibit the issue that you could share with us?

[gaia]

@romac hermes.zip