besu icon indicating copy to clipboard operation
besu copied to clipboard

Intermittent 400 Bad Request when sending a transaction

Open samsondav opened this issue 2 years ago • 5 comments

We are seeing intermittent errors trying to send transactions on Besu (consensus client - teku). It only occurs on some RPC nodes and not all the time.

We are using eth_sendRawTransaction call.

Occasionally, we see this response:

400 Bad Request: {\n \"jsonrpc\" : \"2.0\",\n \"id\" : 60246,\n \"error\" : {\n \"code\" : -32602,\n \"message\" : \"Invalid params\"\n }\n}

This is a blocker for us to run Chainlink on Besu. We need reliable transaction sending.

Could be related to this issue? https://github.com/hyperledger/besu/issues/2251

samsondav avatar Aug 03 '22 15:08 samsondav

Can you provide an example call please? A curl statement would be fine.

jflo avatar Aug 08 '22 17:08 jflo

I'm going to play go-between here. "Literally just send any transaction" ... I assume if you just send enough tx, do include those that call contracts, you'll see the issue.

yorickdowne avatar Aug 16 '22 21:08 yorickdowne

In response to the above situation, this happens to my program from time to time, but after I use the metamask wallet to send data once, my original program can work again. (my program works the same as metamask)

lyxer123 avatar Aug 29 '22 06:08 lyxer123

I use 4G wireless module to post the eth_sendRawTransaction:this error {\n "jsonrpc" : "2.0",\n "id" : 60246,\n "error" : {\n "code" : -32602,\n "message" : "Invalid params"\n }\n} occured many times。

when error occured,I use metamask to send a Transaction using same wallet address,then the 4G wireless module can send data to besu with the same wallet address.

lyxer123 avatar Sep 01 '22 06:09 lyxer123

Hi @samsondav we are taking a look at RPC issues this sprint. Are you able to provide any more context re the intermittency of this issue - anything unusual about how you're sending the transactions in general, or the ones that fail?

We're also working on adding more info to the "invalid params" error logging so hopefully that may assist #4326

macfarla avatar Oct 10 '22 00:10 macfarla

Don't have any more context, it seems completely random, but if you send enough tx you will see it.

samsondav avatar Oct 31 '22 15:10 samsondav

I am also seeing this behavior intermittently on our private 4 node QBFT network. We load test ~10,000 transactions of the same contract interaction type, and we see about 4 of these 400 Invalid params error returned.

"rawTx": {
    "type": "0x0",
    "nonce": "0x5f32",
    "gasPrice": "0x3e8",
    "maxPriorityFeePerGas": null,
    "maxFeePerGas": null,
    "gas": "0x271b8",
    "value": "0x0",
    "input": "0xb660892c00000000000000000000000076113240ed0f339d9552357d10270b7247c452050000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002463656562313766622d653466302d343838642d616261622d616538636265353139356464000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "v": "0x0",
    "r": "0x0",
    "s": "0x0",
    "to": "0x68c30c87759db789f798cbd7689f6107250a572a",
    "hash": "0xec5832268822547cd2de314240fae26be88951754b63f03e06cd02e7d609fba5"
  },

  "error": "400 Bad Request: {\n  \"jsonrpc\" : \"2.0\",\n  \"id\" : 8240,\n  \"error\" : {\n    \"code\" : -32602,\n    \"message\" : \"Invalid params\"\n  }\n}",
  

We are running Besu v10.22.0

baxterjfinch avatar Dec 14 '22 17:12 baxterjfinch

This is still a blocker for Chainlink. We cannot use Besu because of this bug. To reproduce, send literally any transaction with a contract call and do it over and over again. Eventually you will hit this.

samsondav avatar Jan 17 '23 16:01 samsondav

@samsondav how are you sending the multiple transactions?

I tried to reproduce with a simple get/set contract, sending set transaction repeatedly, incrementing the nonce and signing, sending via RPC batch request. Somewhere around 600-800 I hit the "Transaction nonce is too distant from current sender nonce" - which admittedly we only introduced more recently.

macfarla avatar Jan 18 '23 02:01 macfarla

Also - in #4510 we added more detail at DEBUG level for Invalid params. If you can turn logging up to debug (even just for jsonrpc) and send us the log - that would help us diagnose.

macfarla avatar Jan 18 '23 02:01 macfarla

so you would get a message with the invalid call eg 2023-01-18 12:17:15.406+10:00 | vert.x-worker-thread-15 | DEBUG | JsonRpcExecutor | JSON-RPC request -> eth_call [{"data":"0x6d4ce63clkj","to":"0x9a3DBCa554e9f6b9257aAa24010DA8377C57c17e","gas":"0x2faf080"},"latest",null]

macfarla avatar Jan 18 '23 02:01 macfarla

please note with https://github.com/hyperledger/besu/pull/4965 the default max batch size changes to 1, you can increase this using the CLI option if required.

macfarla avatar Jan 23 '23 22:01 macfarla

@macfarla - ok to close?

non-fungible-nelson avatar Jan 26 '23 12:01 non-fungible-nelson

@samsondav Could you try to reproduce it with the latest release that's coming out soon? V.23.1.0.?

macfarla avatar Feb 03 '23 02:02 macfarla

Same here. The problem we noticed is that is intermitent.. so its really hard to reproduce.

It happens maybe 1 tx very 5000 txs. We cannot identify the root issue.. but sending the same exact transaction after it failed works, so the data itself doesnt looks like the issue.

Example of a transaction that failed: curl -X POST http://localhost:18545 -d '{"jsonrpc":"2.0","method":"eth_sendRawTransaction", "params":["0xf869824501808309ba62944be396d29bcd5af6e43ce1c5155693973f3ca8b080840d52b4168303059ea095b845c2f3d4c02e4249013e6ea7e4caa30b41f7bd6daaa0360ff9f1604b5ad8a05bfccbb6898b354fe950179ec331755e7ad826255ba1034f6974da07197d4dbb"], "id": "1"}'

NOTE: we updated from 22.4.2 into 22.7.7 when we started to obtain this errors randomly.

nano-adhara avatar Feb 06 '23 15:02 nano-adhara

Also seeing this same behavior on a private network, been seeing this for a while now. Upgraded to 23.1.1 just now and can confirm the behavior is still the same:

INVALID_PARAMS is returned a small percentage of the time at high load for eth_sendRawTransaction, most of the time, around the time a new block is cut, but not always (I suspect a race condition). Retrying the request without any modification to the request (but with some small ~100ms delay) typically results in success.

We have been seeing these errors since 22.4.4. Hope that helps.

dmitridr avatar Mar 21 '23 15:03 dmitridr

recent AT fail with invalid params - might be a clue https://app.circleci.com/pipelines/github/hyperledger/besu/22285/workflows/dad5b738-a624-4db5-8a90-1a7c8c304019/jobs/138045/tests - possibly related to exception handling, with invalid params used as default error typoe

macfarla avatar May 28 '23 22:05 macfarla

Closed as completed via #5506 (might not solve the initial problem, but will give more info about the cause of the errors). Curious to see any results on this load level with the new approach here.

non-fungible-nelson avatar Jul 10 '23 23:07 non-fungible-nelson

Further improved by https://github.com/hyperledger/besu/pull/6143

macfarla avatar May 14 '24 03:05 macfarla