SUI sync new nodes broken for "Fork detection"

[jun0tpyrc]

Steps to Reproduce Issue

run recent image , .e.g. mysten/sui-node:ci image

Expected Result

could sync to chain head

Actual Result

broken

System Information

Some examples logs when crash

2024-05-24T05:47:52.547209Z  INFO execution_driver{tx_digest=TransactionDigest(GH15CXeAZ7utw84HWQoiJeeZKmAasFibgPsWJBuVpDTo)}: sui_core::authority: Dumped node state for transaction GH15CXeAZ7utw84HWQoiJeeZKmAasFibgPsWJBuVpDTo to /tmp/GH15CXeAZ7utw84HWQoiJeeZKmAasFibgPsWJBuVpDTo_1716529672520_NODE_DUMP.json
2024-05-24T05:47:52.547241Z ERROR execution_driver{tx_digest=TransactionDigest(GH15CXeAZ7utw84HWQoiJeeZKmAasFibgPsWJBuVpDTo)}: sui_core::authority: fork detected! tx_digest=TransactionDigest(GH15CXeAZ7utw84HWQoiJeeZKmAasFibgPsWJBuVpDTo) expected_effects_digest=TransactionEffectsDigest(Eys2yCn8k6H8s1G8MenBWW93tBsvhq98C4nf3uF9Kebr) actual_effects=V1(TransactionEffectsV1 { status: Success, executed_epoch: 85, gas_used: GasCostSummary { computation_cost: 820000, storage_cost: 12372800, storage_rebate: 15032952, non_refundable_storage_fee: 151848 }, modified_at_versions: [(0x1a2a448697679f553d4b8fd27d5cb18250476a46f5b6c62c00a34efade28c43e, SequenceNumber(1888260)), (0x2dd8bed1ad409c6a9065b69dfd4e85e0eb61b8d6f741be5771826095318bffd4, SequenceNumber(7008552)), (0x56bc42f37a89eefc06de2306d4a81442b367efa35ffc351e9bc79c063eee5785, SequenceNumber(1888260)), (0x955e00d17dfb474d8df498f73b5c2b6338da4e8df01292c7113a2064ae43c5d1, SequenceNumber(1888261)), (0xf0c8e045496bddbef8261cd816f21f84368adafee230fa909a2403c473bdbee7, SequenceNumber(7008552)), (0x1172772efa46dc843b903941c439caf28d377a56a7a7e297b8e43c56920957d8, SequenceNumber(1888260))], shared_objects: [(0x64168ef7953cbdb3cf0b3e4f13301061740d2b1d015900e1ae025d31515ad830, SequenceNumber(1679816), o#Guxg1d32nWrMancSnL2yMhfJpGsxVXni4JNnf8sM27qE), (0x2dd8bed1ad409c6a9065b69dfd4e85e0eb61b8d6f741be5771826095318bffd4, SequenceNumber(7008552), o#BYgUnmtb4CRTCq5gK1oeZ36FeJ8z2ZbPsE7iYqYqVXB9), (0x0000000000000000000000000000000000000000000000000000000000000006, SequenceNumber(7018205), o#HRz1nXbVtUCuRSHmJEs3p4LHgELGT9qdrizhNE4Cuwt1), (0xf0c8e045496bddbef8261cd816f21f84368adafee230fa909a2403c473bdbee7, SequenceNumber(7008552), o#2qv9tZTyABTSJfyERSqsMdQkxC9FaHAw7fQDKZ7CYxXG)], transaction_digest: TransactionDigest(GH15CXeAZ7utw84HWQoiJeeZKmAasFibgPsWJBuVpDTo), created: [((0x30014b1624ccfe909d0ada1bfec75cac26bc94556d74d323ea3409e14a2072cf, SequenceNumber(7018206), o#7qHLNDQ8KUE6kLQzJ25V1xej6hT91GQnUbGQEGWKhj7o), AddressOwner(0x19dd56c4b58432a3a1b406a29a2b137715035566c2bb6c450f9731a7054ebacd))], mutated: [((0x1a2a448697679f553d4b8fd27d5cb18250476a46f5b6c62c00a34efade28c43e, SequenceNumber(7018206), o#GQRAg6GbMauntwQHgKUorgiKUW964RWzwNJGhevJB1Sg), ObjectOwner(0x0921dc0e56aa8c1e8ada5bd1743e3bbcfcee824190e93655bdc858322082c68a)), ((0x2dd8bed1ad409c6a9065b69dfd4e85e0eb61b8d6f741be5771826095318bffd4, SequenceNumber(7018206), o#HETU3EaEunC1koHMVTxuKPso721zTPihjkvdhEuBYdVG), Shared { initial_shared_version: SequenceNumber(73) }), ((0x56bc42f37a89eefc06de2306d4a81442b367efa35ffc351e9bc79c063eee5785, SequenceNumber(7018206), o#9cZjjAuJpK36uQtLvEtysZW4eCiCRmKMAornkgThfLh1), ObjectOwner(0x5551bf73117e07a0e009ae6264df55eb1e6825aaaf383d3cb984c494d42c0b8d)), ((0x955e00d17dfb474d8df498f73b5c2b6338da4e8df01292c7113a2064ae43c5d1, SequenceNumber(7018206), o#7DdCDahbUR7LN89Ew5fwDk399oBXFf5yUFRAo68KzG2p), AddressOwner(0x19dd56c4b58432a3a1b406a29a2b137715035566c2bb6c450f9731a7054ebacd)), ((0xf0c8e045496bddbef8261cd816f21f84368adafee230fa909a2403c473bdbee7, SequenceNumber(7018206), o#2kmEUtVvpsHgenuEm67kK8wiGHzr9D9twa6ojnNA3WRV), Shared { initial_shared_version: SequenceNumber(72) })], unwrapped: [], deleted: [(0x1172772efa46dc843b903941c439caf28d377a56a7a7e297b8e43c56920957d8, SequenceNumber(7018206), o#7gyGAp71YXQRoxmFBaHxofQXAipvgHyBKPyxmdSJxyvz)], unwrapped_then_deleted: [], wrapped: [], gas_object: ((0x955e00d17dfb474d8df498f73b5c2b6338da4e8df01292c7113a2064ae43c5d1, SequenceNumber(7018206), o#7DdCDahbUR7LN89Ew5fwDk399oBXFf5yUFRAo68KzG2p), AddressOwner(0x19dd56c4b58432a3a1b406a29a2b137715035566c2bb6c450f9731a7054ebacd)), events_digest: None, dependencies: [TransactionDigest(5vYBgsftyYkJ3TsBjUf14bvcw7RJgvnvVtb7FG2pmPaW), TransactionDigest(CVUpiHULLGe7exLk9WuwcFSFuLsc69snYNYx96RMY5kn), TransactionDigest(EPdyWs7QrS2HxTEJSevZb1bJ54iZWcZbTvMzsuyc2txD), TransactionDigest(G72YVYqfZMtRZLARevpkSHRTetxGkAHCRE3k4zUinB4V), TransactionDigest(HbBEg3P1pRb3b1o5d7Cd5ataN6FdzAT2J5FwETXLs6j9)] })

a different run

2024-05-24T00:15:14.401664Z ERROR execution_driver{tx_digest=TransactionDigest(GLkMRea4YzLm8MJMhpp91en5CZsUek2jPkJC3iKDDLnV)}: sui_core::authority: fork detected! tx_digest=TransactionDigest(GLkMRea4YzLm8MJMhpp91en5CZsUek2jPkJC3iKDDLnV) expected_effects_digest=TransactionEffectsDigest(87LsSWfWCucGTEqj4nRADGpT7HW68EqNWYoPuKjkGBDp) actual_effects=V1(TransactionEffectsV1 { status: Success, executed_epoch: 85, gas_used: GasCostSummary { computation_cost: 820000, storage_cost: 12372800, storage_rebate: 15032952, non_refundable_storage_fee: 151848 }, modified_at_versions: [(0x2dd8bed1ad409c6a9065b69dfd4e85e0eb61b8d6f741be5771826095318bffd4, SequenceNumber(7018206)), (0x3e2f399f782e04fdb6a37aa8d38c966d67d535881627bd01b30119484068b10b, SequenceNumber(1856779)), (0x6f09b2b3d8e99578a1bd34a0c32eb73b43d94dc2f174a6e536b4879bf618833d, SequenceNumber(1856779)), (0xeddd4573e2306c8e9a11a35b6317c4f225b8e982bd1d46e2d28f545804c2191e, SequenceNumber(3530684)), (0xf0c8e045496bddbef8261cd816f21f84368adafee230fa909a2403c473bdbee7, SequenceNumber(7018206)), (0x5491d541758b087030fcc03e41d539b509d92fbb78aa932b889c594e4a13e02c, SequenceNumber(1856779))], shared_objects: [(0x64168ef7953cbdb3cf0b3e4f13301061740d2b1d015900e1ae025d31515ad830, SequenceNumber(1679816), o#Guxg1d32nWrMancSnL2yMhfJpGsxVXni4JNnf8sM27qE), (0x2dd8bed1ad409c6a9065b69dfd4e85e0eb61b8d6f741be5771826095318bffd4, SequenceNumber(7018206), o#HETU3EaEunC1koHMVTxuKPso721zTPihjkvdhEuBYdVG), (0x0000000000000000000000000000000000000000000000000000000000000006, SequenceNumber(7064885), o#HNABM6MkuVYMeVUguAxhDrPLCyuDxE4cJ1ktLN7KxXSm), (0xf0c8e045496bddbef8261cd816f21f84368adafee230fa909a2403c473bdbee7, SequenceNumber(7018206), o#2kmEUtVvpsHgenuEm67kK8wiGHzr9D9twa6ojnNA3WRV)], transaction_digest: TransactionDigest(GLkMRea4YzLm8MJMhpp91en5CZsUek2jPkJC3iKDDLnV), created: [((0x0fd119e4f311a41252760bed5c1445a0aabb1a3979b6dbafd71c80fad80f5adc, SequenceNumber(7064886), o#3JBmSXML4cdfdyQQNRHhWUJZK3sbKwoJfRd2MmYgszQE), AddressOwner(0xad7241270639d7ccfc35b225a521df53b467a066be5ac497a5885799b6411a82))], mutated: [((0x2dd8bed1ad409c6a9065b69dfd4e85e0eb61b8d6f741be5771826095318bffd4, SequenceNumber(7064886), o#49MiLT5K6yw79CoBt7gjzX4ab27tspr5VrvS28eC8MPA), Shared { initial_shared_version: SequenceNumber(73) }), ((0x3e2f399f782e04fdb6a37aa8d38c966d67d535881627bd01b30119484068b10b, SequenceNumber(7064886), o#DywEf7cNMaEhTSXvZcTDsumEhBqXMjLJqNo4SJbVLjLo), ObjectOwner(0x0921dc0e56aa8c1e8ada5bd1743e3bbcfcee824190e93655bdc858322082c68a)), ((0x6f09b2b3d8e99578a1bd34a0c32eb73b43d94dc2f174a6e536b4879bf618833d, SequenceNumber(7064886), o#EB63TztkFSegDpEUDT1zeJN4BjAca3ZtddRDki7SAERi), ObjectOwner(0x5551bf73117e07a0e009ae6264df55eb1e6825aaaf383d3cb984c494d42c0b8d)), ((0xeddd4573e2306c8e9a11a35b6317c4f225b8e982bd1d46e2d28f545804c2191e, SequenceNumber(7064886), o#D5FL1wtxmbtQx5FcB3cDvD4NTzNzXJszkVVuFnctrvY5), AddressOwner(0xad7241270639d7ccfc35b225a521df53b467a066be5ac497a5885799b6411a82)), ((0xf0c8e045496bddbef8261cd816f21f84368adafee230fa909a2403c473bdbee7, SequenceNumber(7064886), o#2ttbpzPMPnoBm3MQrpx3sHNYx2CCubqYWeMwqPxauo7Q), Shared { initial_shared_version: SequenceNumber(72) })], unwrapped: [], deleted: [(0x5491d541758b087030fcc03e41d539b509d92fbb78aa932b889c594e4a13e02c, SequenceNumber(7064886), o#7gyGAp71YXQRoxmFBaHxofQXAipvgHyBKPyxmdSJxyvz)], unwrapped_then_deleted: [], wrapped: [], gas_object: ((0xeddd4573e2306c8e9a11a35b6317c4f225b8e982bd1d46e2d28f545804c2191e, SequenceNumber(7064886), o#D5FL1wtxmbtQx5FcB3cDvD4NTzNzXJszkVVuFnctrvY5), AddressOwner(0xad7241270639d7ccfc35b225a521df53b467a066be5ac497a5885799b6411a82)), events_digest: None, dependencies: [TransactionDigest(4g8wZg3gdn4JxZ1M71EzLPv5zDLsbDKfN83FPvMk35N6), TransactionDigest(9ZR5gsPCbJPywPkcUiCHGgR6DiTnfAtcUPZhqG8y1DXV), TransactionDigest(G72YVYqfZMtRZLARevpkSHRTetxGkAHCRE3k4zUinB4V), TransactionDigest(GH15CXeAZ7utw84HWQoiJeeZKmAasFibgPsWJBuVpDTo), TransactionDigest(HbBEg3P1pRb3b1o5d7Cd5ataN6FdzAT2J5FwETXLs6j9)] })
2024-05-24T00:15:14.401785Z ERROR execution_driver{tx_digest=TransactionDigest(GLkMRea4YzLm8MJMhpp91en5CZsUek2jPkJC3iKDDLnV)}: telemetry_subscribers: panicked at /sui/crates/sui-core/src/authority.rs:1331:17:

attached is one of the dump json of one of the hundred times the nodes crashed (may not being the two above) GLkMRea4YzLm8MJMhpp91en5CZsUek2jPkJC3iKDDLnV_1716509463134_NODE_DUMP.json

[bmwill]

Thanks @jun0tpyrc for the report, we've also noticed this and are investigating.