Devnet finalization rapidly falls behind head

[Arachnid]

Describe the bug When running the op devnet, the latest block proposal on L1 falls farther and farther behind the L2 head.

To Reproduce Steps to reproduce the behavior:

1. Follow the directions at https://community.optimism.io/docs/developers/build/dev-node/ to start up a devnet.
2. Wait an hour or so
3. Check L2OutputOracle.latestBlockNumber() vs eth_getBlock on L2

Expected behavior The latest block number on L1 should be close to the L2 head.

[tynes]

If it falls behind then it must take some time because this is logged in CI

See Test the Stack, copying the logs here:

Waiting to be able to prove withdrawal
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 36
L1 chain tip: 44
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 36
L1 chain tip: 45
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 42
L1 chain tip: 46
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 42
L1 chain tip: 47
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 42
L1 chain tip: 48
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 42
L1 chain tip: 49
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 42
L1 chain tip: 50
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 42
L1 chain tip: 51
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 48
L1 chain tip: 52
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 48
L1 chain tip: 53
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 48
L1 chain tip: 54
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 48
L1 chain tip: 55
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 48
L1 chain tip: 56
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 48
L1 chain tip: 57
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 54
L1 chain tip: 58
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 54
L1 chain tip: 59
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 54
L1 chain tip: 60
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 54
L1 chain tip: 61
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 54
L1 chain tip: 62
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 54
L1 chain tip: 63
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 60
L1 chain tip: 64
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 60
L1 chain tip: 65
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 60
L1 chain tip: 66
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 60
L1 chain tip: 67
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 60
L1 chain tip: 68
Message status: STATE_ROOT_NOT_PUBLISHED
Latest L2OutputOracle commitment number: 60
L1 chain tip: 69
Message status: READY_TO_PROVE
Latest L2OutputOracle commitment number: 66
L1 chain tip: 70
Message status: READY_TO_PROVE
Latest L2OutputOracle commitment number: 66
L1 chain tip: 71

You can see that it stays at most 9 blocks away, perhaps the issue is that the proposer ran out of funds?

[Arachnid]

New blocks continue to be proposed, just a lot slower than they're generated.

If you follow the reproduction instructions you can see this for yourself.

[tynes]

Can confirm that this is an issue

[protolambda]

The deploy config (the template that then becomes the config) has finalizationPeriodSeconds: 2, while the L1 time is 3s, and L2 time is 2s. The proposer is synchronously proposing and confirming, but will fall behind when it has to do that for every L2 block, faster than it can do a L1 tx-confirmation cycle.

To solve, we need to bump finalizationPeriodSeconds in the deploy config to something higher than L1_block_time * proposer_tx_confirmations + some_margin.

Or, we can refactor the proposer for concurrent proposals to keep up with a high throughput, but I'd rather not introduce unnecessarily complexity.

[karlb]

I also see this problem. However, increasing the finalizationPeriodSeconds didn't reduce the lag between L2 block number and the one seen in L2OutputOracle's latestBlockNumber.

The proposed L2 block number is always slightly behind the L1 block number. Coincidence, or is there a reason why the devnet config consistently leads to this result for me?

L1 number               3
L2 number               0
L2OO 0
L1 number               20
L2 number               31
L2OO 12
L1 number               37
L2 number               56
L2OO 30
L1 number               54
L2 number               81
L2OO 48
L1 number               71
L2 number               107
L2OO 66
L1 number               87
L2 number               132
L2OO 78
L1 number               104
L2 number               157
L2OO 96
L1 number               121
L2 number               183
L2OO 114
L1 number               138
L2 number               208
L2OO 132

https://github.com/ethereum-optimism/optimism/issues/5507#issuecomment-1692737606 might be related.

[karlb]

Increasing l2OutputOracleSubmissionInterval solves the problem for me.

[protolambda]

Ah yes, I had the wrong variable; finalizationPeriodSeconds configures how long it takes for an onchain output to become "final" (i.e. what would be 7 days in real networks). l2OutputOracleSubmissionInterval is what changes the interval between outputs (1 hour on real networks). With the batcher submitting blocks, the op-node deriving from that to mark things safe, and the op-proposer only proposing after confirming previous work & safety of blocks, it takes it a little while to finish its proposal cycle. The l2OutputOracleSubmissionInterval has to accomodate for that, so the op-proposer can keep up.

[smartcontracts]

Was this fixed? @hamdiallam

[tynes]

This is a known issue and is still the case when running a pre fault proofs system. A post fault proofs system does not have this issue in theory, because there is not an interval at which outputs must be proposed in. Any output can be proposed anytime

[ajsutton]

Given the devnet now defaults to using fault proofs, I'm going to close this as fixed, but please feel free to reopen if I've misunderstood something.