Transport RPC unfairness when call() is mixed with query() in multithread

[domohawk]

TLDR: query()s on the same Web3<Http> contract seem extremely parallelizable in multithread. But if you add one thread that does a call(), then all sorts of weird slowness happens that smells like locking unfairness.

A bit more details:

I have a 9-thread pool doing evm-intensive query()s, which together take ~1s (individually ~500ms). If I add another 1-thread pool that does a cheap call() to update the contract (individually ~5ms) and it uses the same transport via web3.clone(), then progress gets very poor and unpredictable. One or the other pool are blocked for indeterminate amounts of time. The 9-thread pool query() regularly takes 30s+. The requests don't even panic after the 5s rpc timeout, which makes me think they're queued in the transport.

Both pools sleep for a few seconds between batch requests, and eth.pendingTransactions is emptying quickly.

If I create a separate http transport for the multi- query() and single- call() pools, then things seem to work just fine.

This is tested against geth 1.10.7.