`author_submitAndWatchExtrinsic` can return `Finalized`/`InBlock` even though the tx-hash of the submission is not in the specified block

[AlastairHolmes]

Is there an existing issue?

• [X] I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• [X] This is not a support question.

Description of bug

When submitting signed extrinsics using the rpc author_submitAndWatchExtrinsic we found that sometimes it will return InBlock(block_hash) or Finalized(block_hash), but if you queried the returned block_hash the submitted extrinsic does not exist in that block.

This seems like a bug, as I would expect the author_submitAndWatchExtrinsic to only return InBlock/Finalized if the particular submission I made was the one that was included in the block i.e. the tx-hash matched the tx-hash of what I submitted. Also I am aware that tx-hashes aren't unique in Substrate, I still feel this behavior is peculiar.

Steps to reproduce

Start two nodes. Once the nodes have successfully started up, for each node start a script that connects to that node, and serially submits the same signed extrinsic multiple times (I tested with 20 times), each time using the rpcs system_accountNextIndex (To choose a nonce) and author_submitAndWatchExtrinsic. For each submission calculate and record the txhash of the extrinsic (Note author_submitAndWatchExtrinsic doesn't return the hash itself, so you have to calculate it yourself).

In the script, after it has submitted all the extrinsics. For each TransactionStatus stream returned by author_submitAndWatchExtrinsic, when it returns InBlock(block_hash): Check if the txhash you calculated in association with the submission that created this TransactionStatus stream, exists in the block specified by InBlock(block_hash).

This check fails for atleast a few of the submissions every time within around 3 seconds. But if you only use a single node, or both scripts connect to the same node, it doesn't fail.

Note: The timing of the submissions does seem important, i.e. that both scripts are submitting at the same time. I'm running both nodes and scripts on the same machine, and the party that raised the issue with us, also runs their nodes all in the same data center. Which I imagine makes it easier to create this situation, due to the very small network latency.

We are using the polkadot-v1.6.0 tag (481165d92297d7dfd5eaf9c7f442441761fc0a12) of the polkadot-sdk repo.

[AlastairHolmes]

I found this also occurs if instead of checking the InBlock response, you check using the Finalized response.

I.e. the TransactionStatus stream tells you an extrinsic is finalized in a block, but it doesn't actually exist in that block.

[bkchr]

If you have a script that reproduces this, could you please share it?