besu icon indicating copy to clipboard operation
besu copied to clipboard
verified publisher icon hyperledger

Besu returns empty trace on block 21030627

Open dylanbutler1 opened this issue 1 year ago • 1 comments

Actual behavior Execute curl -H "Content-Type: application/json" -d '{"id": 1,"jsonrpc": "2.0","method": "trace_block","params":["0x140e6e3"]}' localhost:8545

Result: {"jsonrpc":"2.0","id":1,"result":[]}

Expected behavior Expected response is a list of traces.

cc @joshuafernandes

dylanbutler1 avatar Oct 24 '24 01:10 dylanbutler1

@dylanbutler1 Which besu node are you using to test ? You need to have a Besu forest archive node otherwise you will get an empty response as a Besu Bonsai full node can trace by default chain height - 512 blocks. You can test again and if you see this log Limit of historical layers that can be loaded with BONSAI ... It means that you're using Bonsai and it can not perform a trace_block call on a block that is older than chain height - 512

ahamlat avatar Oct 24 '24 09:10 ahamlat

For visibility, there was a similar issue affecting Erigon as well

I am not sure how to resolve this at the moment.

racket2000 avatar Oct 24 '24 19:10 racket2000

This is expected behaviour for bonsai (default). You need to have a Besu forest archive node otherwise you will get an empty response as a Besu Bonsai full node can trace by default chain height - 512 blocks. You can test again and if you see this log Limit of historical layers that can be loaded with BONSAI ... It means that you're using Bonsai and it can not perform a trace_block call on a block that is older than chain height - 512

macfarla avatar Oct 24 '24 23:10 macfarla

I tested with set_newHead on a bonsai node, to change the head near the block we want to trace, but it didn't work, the node had few issues


{"@timestamp":"2024-10-25T19:06:55,322","level":"WARN","thread":"vert.x-worker-thread-3","class":"DefaultBlockchain","message":"Chain Reorganization +0 new / -13956 old
       Old - hash: 0x48a7ab9c62ce82c50c2809475c6ff7077a9577a889e53517991b54ba98785b00, height: 21044611
       New - hash: 0x7bc8e8f0f548333aae54bc5a061b0dc02d931221ac13d6f0e3278aacc25e94e2, height: 21030655
  Ancestor - hash: 0x7bc8e8f0f548333aae54bc5a061b0dc02d931221ac13d6f0e3278aacc25e94e2, height: 21030655","throwable":""}
{"@timestamp":"2024-10-25T19:08:01,179","level":"WARN","thread":"vert.x-eventloop-thread-4","class":"EngineQosTimer","message":"Execution engine not called in 120 seconds, consensus client may not be connected","throwable":""}
{"@timestamp":"2024-10-25T19:08:09,504","level":"INFO","thread":"vert.x-worker-thread-7","class":"DiffBasedWorldStateProvider","message":"Archive rolling failed for block hash 0xc8758a6bd1b3c1184ceaed1e451f3380698b6114c842f45a1ce9f3cedd41f24c","throwable":" java.util.NoSuchElementException: No value present\n\tat java.base/java.util.Optional.get(Optional.java:143)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.rollMutableStateToBlockHash(DiffBasedWorldStateProvider.java:177)\n\
org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.lambda$getMutable$4(DiffBasedWorldStateProvider.java:145)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.getMutable(DiffBasedWorldStateProvider.java:145)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$getAndMapWorldState$24(BlockchainQueries.java:948)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.getAndMapWorldState(BlockchainQueries.java:946)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.lambda$processTracing$1(Tracer.java:49)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.processTracing(Tracer.java:47)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.traceBlock(TraceBlock.java:99)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.lambda$resultByBlockNumber$0(TraceBlock.java:86)\n\tat java.base/java.util.Optional.map(Optional.java:260)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.resultByBlockNumber(TraceBlock.java:86)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.findResultByParamType(AbstractBlockParameterMethod.java:88)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.response(AbstractBlockParameterMethod.java:102)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)\n\tat org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)\n\tat java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)\n\tat io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)\n\tat io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)\n\tat io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)\n\tat io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}
{"@timestamp":"2024-10-25T19:08:09,505","level":"ERROR","thread":"vert.x-worker-thread-7","class":"BlockchainQueries","message":"failed worldstate query for 0xc8758a6bd1b3c1184ceaed1e451f3380698b6114c842f45a1ce9f3cedd41f24c","throwable":" org.hyperledger.besu.ethereum.trie.MerkleTrieException: invalid\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.rollMutableStateToBlockHash(DiffBasedWorldStateProvider.java:249)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.lambda$getMutable$4(DiffBasedWorldStateProvider.java:145)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.getMutable(DiffBasedWorldStateProvider.java:145)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$getAndMapWorldState$24(BlockchainQueries.java:948)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.getAndMapWorldState(BlockchainQueries.java:946)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.lambda$processTracing$1(Tracer.java:49)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.processTracing(Tracer.java:47)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.traceBlock(TraceBlock.java:99)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.lambda$resultByBlockNumber$0(TraceBlock.java:86)\n\tat java.base/java.util.Optional.map(Optional.java:260)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.resultByBlockNumber(TraceBlock.java:86)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.findResultByParamType(AbstractBlockParameterMethod.java:88)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.response(AbstractBlockParameterMethod.java:102)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)\n\tat org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)\n\tat java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)\n\tat io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)\n\tat io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)\n\tat io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)\n\tat io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}

cc. @garyschulte

ahamlat avatar Oct 25 '24 19:10 ahamlat

#7821 adds the capability to move the worldstate along with the blockchain, to a specified block number. Since bonsai is not intended to serve as an archive node, this is a workaround that can be used until bonsai-archive feature is complete.

This should allow a trace of block 21030627 on nodes which have the trielog present (nodes which are not pruning their bonsai history), but it can be a foot-gun if used carelessly or incorrectly.

garyschulte avatar Oct 28 '24 21:10 garyschulte

moving to review for reporter feedback.

jflo avatar Nov 05 '24 15:11 jflo

circling back to close this issue, with debug_setHead from #7821 besu is able to get this trace.

trace attached: trace_block_21030627.json

garyschulte avatar Nov 22 '24 16:11 garyschulte