debug_traceBlockByNumber returning various errors

[gflow33]

What version of iotex-core image (or code branch) are you using?

v2.3.0-d3c7dca

What operating system and processor architecture are you using?

Ubuntu 22.04 amd64

What did you do? If possible, provide a recipe for reproducing the error.

When calling debug_traceBlockByNumber on certain block numbers it returns errors in the logs and response.

What did you expect to see?

No errors

What did you see instead?

'content-type: application/json' -d '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["0x17f2a83",{"tracer":"callTracer","timeout":"30s"}],"id":0}

Nov 03 15:46:30 juju-9f41bc-0 iotex-server[1291035]: {"level":"panic","ts":"2025-11-03T15:46:30.197Z","caller":"evm/evmstatedbadapter.go:449","msg":"Failed to set nonce.","ioAddr":"io1v995up8hnqm3kvswadna6spjkdyfwph4kvecd7","error":"current value 18446744073709551615: nonce overflow","errorVerbose":"nonce overflow\ngithub.com/iotexproject/iotex-core/v2/state.init\n\t<autogenerated>:1\nruntime.doInit1\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7291\nruntime.doInit\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7258\nruntime.main\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:254\nruntime.goexit\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\ncurrent value 18446744073709551615\ngithub.com/iotexproject/iotex-core/v2/state.(*Account).SetPendingNonce\n\t/home/ubuntu/iotex-core/state/account.go:167\ngithub.com/iotexproject/iotex-core/v2/action/protocol/execution/evm.(*StateDBAdapter).SetNonce\n\t/home/ubuntu/iotex-core/action/protocol/execution/evm/evmstatedbadapter.go:448\ngithub.com/iotexproject/iotex-core/v2/action/protocol/execution/evm.(*ErigonStateDBAdapter).SetNonce\n\t/home/ubuntu/iotex-core/action/protocol/execution/evm/evmstatedbadapter_erigon.go:122\ngithub.com/iotexproject/iotex-core/v2/action/protocol/execution/evm.executeInEVM\n\t/home/ubuntu/iotex-core/action/protocol/execution/evm/evm.go:579\ngithub.com/iotexproject/iotex-core/v2/action/protocol/execution/evm.ExecuteContract\n\t/home/ubuntu/iotex-core/action/protocol/execution/evm/evm.go:272\ngithub.com/iotexproject/iotex-core/v2/action/protocol/execution.(*Protocol).Handle\n\t/home/ubuntu/iotex-core/action/protocol/execution/protocol.go:74\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*workingSet).runAction\n\t/home/ubuntu/iotex-core/state/factory/workingset.go:197\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*workingSet).runActionsLegacy\n\t/home/ubuntu/iotex-core/state/factory/workingset.go:662\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*workingSet).processLegacy\n\t/home/ubuntu/iotex-core/state/factory/workingset.go:641\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*workingSet).Process\n\t/home/ubuntu/iotex-core/state/factory/workingset.go:527\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*stateDB).WorkingSetAtTransaction\n\t/home/ubuntu/iotex-core/state/factory/statedb.go:413\ngithub.com/iotexproject/iotex-core/v2/api.(*coreService).traceBlock\n\t/home/ubuntu/iotex-core/api/coreservice.go:2326\ngithub.com/iotexproject/iotex-core/v2/api.(*coreService).TraceBlockByNumber\n\t/home/ubuntu/iotex-core/api/coreservice.go:2138\ngithub.com/iotexproject/iotex-core/v2/api.(*web3Handler).traceBlockByNumber\n\t/home/ubuntu/iotex-core/api/web3server.go:1326\ngithub.com/iotexproject/iotex-core/v2/api.(*web3Handler).handleWeb3Req\n\t/home/ubuntu/iotex-core/api/web3server.go:265\ngithub.com/iotexproject/iotex-core/v2/api.(*web3Handler).HandlePOSTReq\n\t/home/ubuntu/iotex-core/api/web3server.go:132\ngithub.com/iotexproject/iotex-core/v2/api.(*hTTPHandler).ServeHTTP\n\t/home/ubuntu/iotex-core/api/http.go:74\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:171\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:65\nnet/http.HandlerFunc.ServeHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2220\nnet/http.(*ServeMux).ServeHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2747\nnet/http.serverHandler.ServeHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3210\nnet/http.(*conn).serve\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2092\nruntime.goexit\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700","addr":"0xcd7a0d7a165c0a222b938b513766f4b628bfe0d5","pendingNonce":0,"nonce":0,"execution":"7f37a25530e77185f0885837a9321e9be59438ea6c6e45413f8e7123fd8bb316"}
Nov 03 15:46:30 juju-9f41bc-0 iotex-server[1291035]: {"level":"error","ts":"2025-11-03T15:46:30.199Z","caller":"factory/statedb.go:409","msg":"Recovered from panic in WorkingSetAtTransaction","ioAddr":"io1v995up8hnqm3kvswadna6spjkdyfwph4kvecd7","error":"panic occurred while processing actions: Failed to set nonce.","errorVerbose":"panic occurred while processing actions: Failed to set nonce.\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*stateDB).WorkingSetAtTransaction.func1\n\t/home/ubuntu/iotex-core/state/factory/statedb.go:408\nruntime.gopanic\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/panic.go:791\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/go.uber.org/[email protected]/logger.go:285\ngithub.com/uptrace/opentelemetry-go-extra/otelzap.LoggerWithCtx.Panic\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/github.com/uptrace/opentelemetry-go-extra/[email protected]/otelzap.go:311\ngithub.com/iotexproject/iotex-core/v2/action/protocol/execution/evm.(*StateDBAdapter).SetNonce\n\t/home/ubuntu/iotex-core/action/protocol/execution/evm/evmstatedbadapter.go:449\ngithub.com/iotexproject/iotex-core/v2/action/protocol/execution/evm.(*ErigonStateDBAdapter).SetNonce\n\t/home/ubuntu/iotex-core/action/protocol/execution/evm/evmstatedbadapter_erigon.go:122\ngithub.com/iotexproject/iotex-core/v2/action/protocol/execution/evm.executeInEVM\n\t/home/ubuntu/iotex-core/action/protocol/execution/evm/evm.go:579\ngithub.com/iotexproject/iotex-core/v2/action/protocol/execution/evm.ExecuteContract\n\t/home/ubuntu/iotex-core/action/protocol/execution/evm/evm.go:272\ngithub.com/iotexproject/iotex-core/v2/action/protocol/execution.(*Protocol).Handle\n\t/home/ubuntu/iotex-core/action/protocol/execution/protocol.go:74\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*workingSet).runAction\n\t/home/ubuntu/iotex-core/state/factory/workingset.go:197\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*workingSet).runActionsLegacy\n\t/home/ubuntu/iotex-core/state/factory/workingset.go:662\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*workingSet).processLegacy\n\t/home/ubuntu/iotex-core/state/factory/workingset.go:641\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*workingSet).Process\n\t/home/ubuntu/iotex-core/state/factory/workingset.go:527\ngithub.com/iotexproject/iotex-core/v2/state/factory.(*stateDB).WorkingSetAtTransaction\n\t/home/ubuntu/iotex-core/state/factory/statedb.go:413\ngithub.com/iotexproject/iotex-core/v2/api.(*coreService).traceBlock\n\t/home/ubuntu/iotex-core/api/coreservice.go:2326\ngithub.com/iotexproject/iotex-core/v2/api.(*coreService).TraceBlockByNumber\n\t/home/ubuntu/iotex-core/api/coreservice.go:2138\ngithub.com/iotexproject/iotex-core/v2/api.(*web3Handler).traceBlockByNumber\n\t/home/ubuntu/iotex-core/api/web3server.go:1326\ngithub.com/iotexproject/iotex-core/v2/api.(*web3Handler).handleWeb3Req\n\t/home/ubuntu/iotex-core/api/web3server.go:265\ngithub.com/iotexproject/iotex-core/v2/api.(*web3Handler).HandlePOSTReq\n\t/home/ubuntu/iotex-core/api/web3server.go:132\ngithub.com/iotexproject/iotex-core/v2/api.(*hTTPHandler).ServeHTTP\n\t/home/ubuntu/iotex-core/api/http.go:74\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:171\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:65\nnet/http.HandlerFunc.ServeHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2220\nnet/http.(*ServeMux).ServeHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2747\nnet/http.serverHandler.ServeHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3210\nnet/http.(*conn).serve\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2092\nruntime.goexit\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700"}
Nov 03 15:46:30 juju-9f41bc-0 iotex-server[1291035]: {"level":"error","ts":"2025-11-03T15:46:30.199Z","caller":"api/http.go:87","msg":"fail to respond request.","ioAddr":"io1v995up8hnqm3kvswadna6spjkdyfwph4kvecd7","error":"recovered from panic: runtime error: invalid memory address or nil pointer dereference, request params: {\"jsonrpc\":\"2.0\",\"method\":\"debug_traceBlockByNumber\",\"params\":[\"0x17f2a83\",{\"tracer\":\"callTracer\",\"timeout\":\"30s\"}],\"id\":0}: panic","errorVerbose":"panic\ngithub.com/iotexproject/iotex-core/v2/api.init\n\t<autogenerated>:1\nruntime.doInit1\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7291\nruntime.doInit\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7258\nruntime.main\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:254\nruntime.goexit\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\nrecovered from panic: runtime error: invalid memory address or nil pointer dereference, request params: {\"jsonrpc\":\"2.0\",\"method\":\"debug_traceBlockByNumber\",\"params\":[\"0x17f2a83\",{\"tracer\":\"callTracer\",\"timeout\":\"30s\"}],\"id\":0}\ngithub.com/iotexproject/iotex-core/v2/api.(*web3Handler).HandlePOSTReq.func1\n\t/home/ubuntu/iotex-core/api/web3server.go:124\nruntime.gopanic\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/panic.go:791\nruntime.panicmem\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/panic.go:262\nruntime.sigpanic\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/signal_unix.go:917\ngithub.com/iotexproject/iotex-core/v2/api.(*coreService).traceBlock\n\t/home/ubuntu/iotex-core/api/coreservice.go:2330\ngithub.com/iotexproject/iotex-core/v2/api.(*coreService).TraceBlockByNumber\n\t/home/ubuntu/iotex-core/api/coreservice.go:2138\ngithub.com/iotexproject/iotex-core/v2/api.(*web3Handler).traceBlockByNumber\n\t/home/ubuntu/iotex-core/api/web3server.go:1326\ngithub.com/iotexproject/iotex-core/v2/api.(*web3Handler).handleWeb3Req\n\t/home/ubuntu/iotex-core/api/web3server.go:265\ngithub.com/iotexproject/iotex-core/v2/api.(*web3Handler).HandlePOSTReq\n\t/home/ubuntu/iotex-core/api/web3server.go:132\ngithub.com/iotexproject/iotex-core/v2/api.(*hTTPHandler).ServeHTTP\n\t/home/ubuntu/iotex-core/api/http.go:74\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:171\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:65\nnet/http.HandlerFunc.ServeHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2220\nnet/http.(*ServeMux).ServeHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2747\nnet/http.serverHandler.ServeHTTP\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3210\nnet/http.(*conn).serve\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2092\nruntime.goexit\n\t/home/ubuntu/.gvm/pkgsets/go1.21.8/global/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700"}

'content-type: application/json' -d '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["0x248c462",{"tracer":"callTracer","timeout":"30s"}],"id":0}'

{"jsonrpc":"2.0","id":0,"error":{"code":-32603,"message":"nonce too low"}}

'content-type: application/json' -d '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["0x5281e1",{"tracer":"callTracer","timeout":"30s"}],"id":0}

{"jsonrpc":"2.0","id":0,"error":{"code":-32603,"message":"error when run action: error when action 67d4a795a3180abeaf80165d14124e470d0f1ad4da109a72e276b030413144a1 mutates states: failed to deposit gas: failed to get state of ns = 4163636f756e74 and key = 72e48afbc999a7f892ebaf3543446469acbcd1a0: state does not exist"}}