besu
besu copied to clipboard
hyperledger
Arbitrum warnings. It expects `required field 'v' in transaction`.
Description
As an Abritrum node operator, I want an error and warning free log so that I can focus on the application. I am using Besu as my execution engine and I am seeing the following in my arbitrum logs:
WARN [03-19|15:27:23.581] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-19|15:27:23.673] created block l2Block=192,032,528 l2BlockHash=ff2627..bbeaf9
INFO [03-19|15:27:24.624] rpc response method=eth_getTransactionByBlockHashAndIndex logId=4,501,040 err="missing required field 'v' in transaction" result={} attempt=0 args="[\"0x3a3a66ee3f8d4230bc56eba47d07f466864312e84d967b653644d2071f68079e\", \"0x56\"]"
Transactions are being submitted so far, however these messages are a concern.
Acceptance Criteria
The issue is resolved so that the warning and error messages are not logged in Arbitrum. If this is no longer a required field and is an Arbitrum issue, let me know and I'll open an issue on their side.
Steps to Reproduce (Bug)
- Set up a full node with Besu as the execution engine and Teku as the consensus client.
- Set up an arbitrum node to use the node in step one.
- Let the arbitrum node sync and watch the logs.
Expected behavior: [What you expect to happen]
Actual behavior: [What actually happens]
Frequency: [What percentage of the time does it occur?]
Logs (if a bug)
Please post relevant logs from Besu (and the consensus client, if running proof of stake) from before and after the issue.
Versions (Add all that apply)
- Software version: [
besu --version] 24.3.0 Container tag hyperleger/besu:24.3.0 - Java version: [
java -version] - OS Name & Version: [
cat /etc/*release]
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
- Kernel Version: [
uname -a]
Linux rocketpool-3 5.15.0-100-generic #110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
- Virtual Machine software & version: [
vmware -v] Docker - Docker Version: [
docker version]
Docker version 25.0.4, build 1a576c5
- Cloud VM, type, size: [Amazon Web Services I3-large] Hetzner AX52 #2323003
- Consensus Client & Version if using Proof of Stake: [e.g. Teku, Lighthouse, Prysm, Nimbus, Lodestar]
Teku
consensys/teku:24.3.0
Additional Information (Add any of the following or anything else that may be relevant)
- Besu setup info - genesis file, config options - Running Besu in rocketpool
- System info - memory, CPU AMD Ryzen 7 7700 Octa-Core "Raphael" (Zen4) 64 GB DDR5 RAM
I'm experiencing the same while using consensys/teku:24.3.0 & hyperledger/besu:24.3.0
Please investigate the issue since it causes troubles with Arbitrum node.
Description
After Decun upgrade, local Arbitrum node using Besu client throws a lot of warnings for missing a field. This does not happen with Geth client. The connection may flood the network somehow. This did not happen before the Decun upgrade. . Arbitrum node cannot be synced correctly.
Steps to Reproduce (Bug)
- Run Besu node
- Run Arbitrum node using Besu as L1
Expected behavior: The node should run smoothly without any continuously throwing warnings
Actual behavior: Warnings and reconnection. Arbitrum node cannot be synced correctly
Frequency: Always
Logs (if a bug)
Besu is running smoothly. No issue from Besu client. Arbitrum client log
WARN [03-26|15:36:47.929] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:48.939] rpc response method=eth_getTransactionByBlockHashAndIndex logId=929 err="missing required field 'v' in transaction" result={} attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:48.939] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:48.964] created block l2Block=194,447,134 l2BlockHash=5c210b..9aa6e6
INFO [03-26|15:36:49.956] rpc response method=eth_getTransactionByBlockHashAndIndex logId=938 err="missing required field 'v' in transaction" result={} attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:49.956] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:49.964] created block l2Block=194,447,146 l2BlockHash=e33e97..822182
INFO [03-26|15:36:50.965] created block l2Block=194,447,150 l2BlockHash=89de18..a32da7
INFO [03-26|15:36:50.969] rpc response method=eth_getTransactionByBlockHashAndIndex logId=947 err="missing required field 'v' in transaction" result={} attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:50.969] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:51.966] created block l2Block=194,447,153 l2BlockHash=1337a2..27d988
INFO [03-26|15:36:51.978] rpc response method=eth_getTransactionByBlockHashAndIndex logId=958 err="missing required field 'v' in transaction" result={} attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:51.978] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:52.966] created block l2Block=194,447,158 l2BlockHash=1e53c0..98604b
INFO [03-26|15:36:52.987] rpc response method=eth_getTransactionByBlockHashAndIndex logId=967 err="missing required field 'v' in transaction" result={} attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:52.987] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:53.966] created block l2Block=194,447,160 l2BlockHash=0ea965..1fb963
INFO [03-26|15:36:54.000] rpc response method=eth_getTransactionByBlockHashAndIndex logId=976 err="missing required field 'v' in transaction" result={} attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:54.000] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:54.967] created block l2Block=194,447,166 l2BlockHash=adf619..9c831e
INFO [03-26|15:36:55.011] rpc response method=eth_getTransactionByBlockHashAndIndex logId=985 err="missing required field 'v' in transaction" result={} attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:55.011] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
Versions (Add all that apply)
- Software version: 24.3.0
- Java version: java 20.0.2
- OS Name & Version: Ubuntu 22.04.1 LTS
- Consensus Client & Version if using Proof of Stake: Teku
The v field is deprecated, and optional, for modern TXes in the RPC. A strong argument can be made this is a bug to ARbitrum as they don't support the current spec.
Type 1 / Access List / EIP-2930 TXes - https://github.com/ethereum/execution-apis/blob/main/src/schemas/transaction.yaml#L293-L296 Type 2 / fee market / EIP-1559 TXes - https://github.com/ethereum/execution-apis/blob/main/src/schemas/transaction.yaml#L268-L271
Type 3 / blob / EIP-4844 TXes don't even have the field in the API spec https://github.com/ethereum/execution-apis/blob/main/src/schemas/transaction.yaml#L237-L252
Working it in #6819 - but I need to get at least one test in before merging.
@ebadiere @dandavid3000 can you try a dev build and see if the patch works?
Docker containers would be much easier for me to test. The tar files will be a bit more time-consuming for me, so if someone else wants to confirm it that would be great. Also, I originally reported that "transactions are being submitted so far", but now my arbitrum node as stopped syncing L2Blocks, with the following in the log:
WARN [03-29|21:25:46.166] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-29|21:25:47.225] rpc response method=eth_getTransactionByBlockHashAndIndex logId=7026 err="missing required field 'v' in transaction" result={} attempt=0 args="[\"0x583f20cae0f372327282fa351e1251a7d71b90a8b09a0fe71935440af6866b52\", \"0x75\"]"
WARN [03-29|21:25:47.225] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-29|21:25:47.362] shutting down because of sigint
INFO [03-29|21:25:47.362] HTTP server stopped endpoint=[::]:8547
INFO [03-29|21:25:47.362] HTTP server stopped endpoint=[::]:8548
INFO [03-29|21:25:47.362] delayed sequencer: context done err="context canceled"
INFO [03-29|21:25:47.363] Waiting background transaction indexer to exit
INFO [03-29|21:25:47.378] Writing cached state to disk block=193,592,630 hash=f7d39e..5da887 root=36c8e2..60d884
INFO [03-29|21:25:47.413] Persisted trie from memory database nodes=14214 size=4.75MiB time=35.277469ms gcnodes=3489 gcsize=1.38MiB gctime=8.566473ms livenodes=23482 livesize=9.10MiB
INFO [03-29|21:25:47.413] Writing cached state to disk block=193,592,629 hash=a97686..43d0a7 root=7f6e31..e124cf
INFO [03-29|21:25:47.414] Persisted trie from memory database nodes=189 size=75.19KiB time="912.887µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=23293 livesize=9.03MiB
INFO [03-29|21:25:47.414] Writing cached state to disk block=193,592,503 hash=d94630..db5ed0 root=f46136..330bd1
INFO [03-29|21:25:47.423] Persisted trie from memory database nodes=1893 size=713.91KiB time=8.752817ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=21400 livesize=8.33MiB
INFO [03-29|21:25:47.423] Writing cached state to disk block=193,592,503 hash=d94630..db5ed0 root=f46136..330bd1
INFO [03-29|21:25:47.423] Persisted trie from memory database nodes=0 size=0.00B time="1.44µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=21400 livesize=8.33MiB
INFO [03-29|21:25:47.423] Writing snapshot state to disk root=f8707d..e937dd
INFO [03-29|21:25:47.423] Persisted trie from memory database nodes=0 size=0.00B time="1.03µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=21400 livesize=8.33MiB
INFO [03-29|21:25:47.457] Blockchain stopped
It could not get passed l2Block: 193,592,630 It seems to be related to me. Can anyone confirm this? Do we expect this fix to resolved in this fix?
I don't think we do dev docker distributions anymore, but @jflo may know better.
From a local repo the command ./gradlew distDocker generates local docker images. Distributing that across non-build machines is left as an exercise for the operator.
On the Arbitrum side, we use the go-ethereum ethclient library to access the parent chain RPC. It seems like we hadn't pulled in https://github.com/ethereum/go-ethereum/pull/28564 . We'll aim to fix that in an upcoming release. Thanks for the report!
I was checking the changed files in the new release https://github.com/hyperledger/besu/releases/tag/24.3.3 but I couldn't see your changes. Am I missing something or does this version not include the solution? @shemnon
I am using 24.3.3 and I am still seeing the same behavior. Am I using the wrong version?
INFO [04-21|23:33:32.612] rpc response method=eth_getTransactionByBlockHashAndIndex logId=1,449,012 err="missing required field 'v' in transaction" result={} attempt=0 args="[\"0x583f20cae0f372327282fa351e1251a7d71b90a8b09a0fe71935440af6866b52\", \"0x75\"]"
WARN [04-21|23:33:32.612] error reading inbox err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
The fix did no make it into 24.3.3, that was a cherry pick release and this one didn't make the cherry pick list.
If the next release is a cherry pick I will endeveor to get this on the cherry pick list.