nethermind icon indicating copy to clipboard operation
nethermind copied to clipboard

Nethermind periodically responds to sendRawTransaction with "fee too low"

Open dceleda opened this issue 2 years ago • 4 comments

Raised by @MicahZoltu on Discord: Nethermind periodically responds to sendRawTransaction with "fee too low" when my fee is way above what should be necessary. 100 max fee, 1 priority fee. If I submit again afterwards it will usually go through. It happens fairly reliably, almost every transaction fails with "fee too low" first, then succeeds by just submitting it again.

dceleda avatar Jul 26 '22 15:07 dceleda

I was not able to reproduce this issue on my local Goerli node. Can you share what is the accompanying message when you get "fee too low" error @MicahZoltu ?

tanishqjasoria avatar Jul 28 '22 10:07 tanishqjasoria

I am now running with tracing enabled every time a fee too low transaction shows up. Unfortunately, I haven't been doing a lot of transactions lately so I haven't had an opportunity to grab the error (it goes away for a while after a node restart). As soon as it happens again I'll post the extra log details here.

MicahZoltu avatar Jul 28 '22 11:07 MicahZoltu

This is a blocker for Chainlink to run on Nethermind

samsondav avatar Aug 08 '22 15:08 samsondav

I am now running with tracing enabled every time a fee too low transaction shows up. Unfortunately, I haven't been doing a lot of transactions lately so I haven't had an opportunity to grab the error (it goes away for a while after a node restart). As soon as it happens again I'll post the extra log details here.

These logs will be extremely helpful. It's very hard to reproduce

marcindsobczak avatar Aug 08 '22 16:08 marcindsobczak

What is the status of the issue?

dceleda avatar Aug 31 '22 16:08 dceleda

I am still running with tracing enabled, but I haven't run into the problem again. This is complicated by the fact that I don't do a lot of transactions, so it may be a while before I reproduce.

MicahZoltu avatar Sep 01 '22 04:09 MicahZoltu

Do we know the status of this issue as of this week?

vyzaldysanchez avatar Sep 06 '22 15:09 vyzaldysanchez

We are not able to reproduce the issue on our end.

tanishqjasoria avatar Sep 07 '22 02:09 tanishqjasoria

I found the original error message in discord history:

{"jsonrpc":"2.0","error":{"code":-32010,"message":"FeeTooLow, FeePerGas needs to be higher than 21000000000 to be added to the TxPool. Affordable FeePerGas of rejected tx: 20676950200."},"id":8}

Based on tx.CalculateAffordableGasPrice() and tx.CalculateEffectiveGasPrice(), it looks like this error should be expected occasionally whenever the tx pool is full and Nethermind doesn't think the transaction will be more valuable than the least valuable transaction it has pending.

In this case, the least valuable transaction was worth tx.GasBottleneck = 21 Gwei per gas.

The estimated "Affordable FeePerGas" (20.7 Gwei) is calculated based on the minimum of 3 potentially limiting factors: MaxGasFee, MaxPriorityGasFee, and the account balance of the Sender. MaxPriorityFeePerGas was 1 Gwei, which means the most value this transaction could have is BaseFee + 1 Gwei. And it could be less than that if the account balance cannot fully cover tx.Value + GasLimit * (BaseFee + 1Gwei).

Assuming a low account balance wasn't the problem, this means the current BaseFee was 19.7 Gwei, and that everyone in the queue had MaxPriorityFeePerGas set to at least 1.3 Gwei.

I'm not sure if it's possible for the base fee used to compute tx.GasBottleneck to be different from the base fee used to compute AffordableGasPrice, but if a slightly higher base fee was used for the first one, and it wasn't updated before this new tx came in, that could explain it too.

reductionista avatar Sep 13 '22 03:09 reductionista

The "fee too low" dropping code path shouldn't be reachable for transactions sent directly to the client over JSON-RPC. Any transactions sent directly to the client should get preferential treatment compared to the rest of the mempool and be retained even if normally they would be kicked out.

You are correct though for transactions received over gossip, they should be kicked out of the mempool if it gets too full.

MicahZoltu avatar Sep 13 '22 09:09 MicahZoltu

Ah, good point--in that case, I wonder if re-orgs could be the problem.

It removes each tx from the the mempool as soon as it receives a new head including it. But then if there's a re-org, it adds it back into the mempool, but in doing so it loses its preferential status as a local transaction: https://github.com/NethermindEth/nethermind/blob/master/src/Nethermind/Nethermind.TxPool/TxPool.cs#L186-L198

Any chance that's what might have happened?

reductionista avatar Sep 14 '22 21:09 reductionista

In my case, the "fee too low" response came in response to sendRawTransaction and sendTransaction calls, which I think wouldn't be subject to reorgs like this because we get the response essentially immediately. If there was a race condition, I would be surprised given the frequency I got the error back when I was getting it regularly. I haven't got it in a while though, so maybe it has been accidentally fixed?

MicahZoltu avatar Sep 16 '22 10:09 MicahZoltu

Chances this is related to #4829 and hence fixed in v1.15.0?

yorickdowne avatar Dec 28 '22 15:12 yorickdowne

Definitely looks like it is the same bug!

MicahZoltu avatar Dec 28 '22 16:12 MicahZoltu