celestia-node icon indicating copy to clipboard operation
celestia-node copied to clipboard

Setting "trusted hash" to older header breaks pruner and prevents startup

Open S1nus opened this issue 1 year ago • 6 comments

Celestia Node version

Commit: 7af07bd08f4eed66fbc9eee47762fd2c3365c89f

OS

MacOS

Install tools

No response

Others

No response

Steps to reproduce it

  1. Set TrustedHash to a new header
  2. start and sync the node
  3. stop the node
  4. set TrustedHash to an older header

Expected result

Node should sync the headers in between the newly set TrustedHash and the oldest stored header

Actual result

2024-12-11T16:53:11.668-0500	ERROR	pruner/service	pruner/service.go:107	failed to get last pruned header	{"height": 1, "err": "header: not found"}
2024-12-11T16:53:11.668-0500	WARN	pruner/service	pruner/service.go:109	exiting pruner service!
2024-12-11T16:53:12.300-0500	INFO	canonical-log	swarm/swarm_dial.go:620	CANONICAL_PEER_STATUS: peer=12D3KooWAAzkuwRBps6qar4Js54E37usSzW57N5QFeDdAVjxt6F1 addr=/ip4/65.21.203.101/tcp/2121 sample_rate=100 connection_status="established" dir="outbound"
panic: invalid type received %!s(<nil>)

goroutine 349 [running]:
github.com/celestiaorg/go-header/p2p.(*subscription[...]).NextHeader(0x10804cc00?, {0x10806e830, 0x140014e0cd0?})
	/Users/redacted/go/pkg/mod/github.com/celestiaorg/[email protected]/p2p/subscription.go:47 +0x2ac
github.com/celestiaorg/celestia-node/share/shwap/p2p/shrex/peers.(*Manager).subscribeHeader(0x1400173a6e0, {0x10806e830, 0x140014e0cd0}, {0x10804c790, 0x14001ba5908})
	/Users/redacted/celestia-node/share/shwap/p2p/shrex/peers/manager.go:302 +0xac
created by github.com/celestiaorg/celestia-node/share/shwap/p2p/shrex/peers.(*Manager).Start in goroutine 198
	/Users/redacted/celestia-node/share/shwap/p2p/shrex/peers/manager.go:166 +0x3e8

Relevant log output

No response

Is the node "stuck"? Has it stopped syncing?

No response

Notes

No response

S1nus avatar Dec 11 '24 21:12 S1nus

Might not be related to pruner, I noticed that warning seems to be there even after I nuked .celestia-light and started fresh

S1nus avatar Dec 11 '24 21:12 S1nus

Hey @S1nus. Thanks for reporting. The fix is here with additional details.

Wondertan avatar Dec 12 '24 07:12 Wondertan

@S1nus you're right though that pruner is not designed to run on a node with a fragmented header-chain. I need to investigate this so @Wondertan 's fix is actually targetting the panic from the subscription, but the pruner will crap out (non-fatally) for a non-contiguous headerchain.

renaynay avatar Dec 12 '24 14:12 renaynay

Yeah, I thought the issue was in the panic that we have to fix anyway independently

Wondertan avatar Dec 12 '24 17:12 Wondertan

@Wondertan Will this be solved by header pruning release?

walldiss avatar Jun 26 '25 12:06 walldiss

Yes

Wondertan avatar Jun 28 '25 03:06 Wondertan