L2IngestionService._syncSequencerBlocks - TypeError: blocks is not iterable

[cgebe]

Describe the bug I am starting the Data Transport Layer with the following env vars:

        - name: DATA_TRANSPORT_LAYER__DB_PATH
          value: "/optimism/dtl"
        - name: DATA_TRANSPORT_LAYER__POLLING_INTERVAL
          value: "1000"
        - name: DATA_TRANSPORT_LAYER__L2_CHAIN_ID
          value: "10"
        - name: DATA_TRANSPORT_LAYER__NODE_ENV
          value: "production"
        - name: DATA_TRANSPORT_LAYER__ETH_NETWORK_NAME
          value: "mainnet"
        - name: DATA_TRANSPORT_LAYER__ADDRESS_MANAGER
          value: "0xdE1FCfB0851916CA5101820A69b13a4E276bd81F"
        - name: DATA_TRANSPORT_LAYER__SERVER_HOSTNAME
          value: "0.0.0.0"
        - name: DATA_TRANSPORT_LAYER__SERVER_PORT
          value: "7878"
        - name: DATA_TRANSPORT_LAYER__SYNC_FROM_L1
          value: "false"
        - name: DATA_TRANSPORT_LAYER__L1_RPC_ENDPOINT
          value: "http://x.x.x.x:3000"
        - name: DATA_TRANSPORT_LAYER__SYNC_FROM_L2
          value: "true"
        - name: DATA_TRANSPORT_LAYER__L2_RPC_ENDPOINT
          value: "https://mainnet.optimism.io"

"http://x.x.x.x:3000" being my ETH node. The service has synced up to current block, but now always crashes after a couple of minutes with the following error:

{"level":30,"time":1662493954598,"msg":"Service is starting..."}
{"level":30,"time":1662493954598,"msg":"Service is initializing..."}
{"level":30,"time":1662493954598,"msg":"Initializing L1 Data Transport Service..."}
{"level":30,"time":1662493954625,"msg":"L2 chain ID is: 10"}
{"level":30,"time":1662493954625,"msg":"BSS HF1 will activate at: 2824317"}
{"level":30,"time":1662493954630,"msg":"Service is initializing..."}
{"level":30,"time":1662493954637,"defaultBackend":"l1","l1GasPriceBackend":"l1","msg":"HTTP Server Options"}
{"level":30,"time":1662493954637,"url":"http://5.9.80.199:31547","msg":"HTTP Server L1 RPC Provider initialized"}
{"level":30,"time":1662493954637,"url":"https://mainnet.optimism.io","msg":"HTTP Server L2 RPC Provider initialized"}
{"level":30,"time":1662493954637,"msg":"Service has initialized."}
{"level":30,"time":1662493954637,"msg":"Service is initializing..."}
{"level":30,"time":1662493954637,"msg":"Service has initialized."}
{"level":30,"time":1662493954637,"msg":"Service has initialized."}
{"level":30,"time":1662493954637,"msg":"Service is starting..."}
{"level":30,"time":1662493954637,"msg":"Service is starting..."}
{"level":30,"time":1662493954638,"host":"0.0.0.0","port":7878,"msg":"Server started and listening"}
{"level":30,"time":1662493954638,"msg":"Service has started"}
{"level":30,"time":1662493955445,"fromBlock":22187460,"toBlock":22187474,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662493957092,"fromBlock":22187474,"toBlock":22187475,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662493958545,"currentL2Block":22187475,"targetL2Block":22187475,"msg":"All Layer 2 (Optimism) transactions are synchronized"}
...
{"level":30,"time":1662494458243,"currentL2Block":22188136,"targetL2Block":22188136,"msg":"All Layer 2 (Optimism) transactions are synchronized"}
{"level":30,"time":1662494459790,"fromBlock":22188136,"toBlock":22188141,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494461218,"fromBlock":22188141,"toBlock":22188150,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494462699,"fromBlock":22188150,"toBlock":22188154,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494464147,"fromBlock":22188154,"toBlock":22188157,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494466033,"currentL2Block":22188157,"targetL2Block":22188157,"msg":"All Layer 2 (Optimism) transactions are synchronized"}
{"level":30,"time":1662494467158,"currentL2Block":22188157,"targetL2Block":22188157,"msg":"All Layer 2 (Optimism) transactions are synchronized"}
{"level":30,"time":1662494468439,"fromBlock":22188157,"toBlock":22188160,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494469883,"fromBlock":22188160,"toBlock":22188163,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494471276,"currentL2Block":22188163,"targetL2Block":22188163,"msg":"All Layer 2 (Optimism) transactions are synchronized"}
{"level":30,"time":1662494472512,"fromBlock":22188163,"toBlock":22188165,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494473982,"fromBlock":22188165,"toBlock":22188169,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494475466,"fromBlock":22188169,"toBlock":22188170,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494477235,"fromBlock":22188170,"toBlock":22188171,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494478584,"currentL2Block":22188171,"targetL2Block":22188171,"msg":"All Layer 2 (Optimism) transactions are synchronized"}
{"level":30,"time":1662494479791,"currentL2Block":22188171,"targetL2Block":22188171,"msg":"All Layer 2 (Optimism) transactions are synchronized"}
{"level":30,"time":1662494480995,"currentL2Block":22188171,"targetL2Block":22188171,"msg":"All Layer 2 (Optimism) transactions are synchronized"}
{"level":30,"time":1662494482223,"currentL2Block":22188171,"targetL2Block":22188171,"msg":"All Layer 2 (Optimism) transactions are synchronized"}
{"level":30,"time":1662494483443,"fromBlock":22188171,"toBlock":22188172,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494484920,"fromBlock":22188172,"toBlock":22188175,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494486810,"fromBlock":22188175,"toBlock":22188177,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494488201,"fromBlock":22188177,"toBlock":22188178,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494489976,"fromBlock":22188178,"toBlock":22188181,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494491377,"fromBlock":22188181,"toBlock":22188183,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
{"level":30,"time":1662494492771,"fromBlock":22188183,"toBlock":22188189,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
Well, that's that. We ran into a fatal error. Here's the dump. Goodbye!
/opt/optimism/packages/data-transport-layer/dist/src/services/l2-ingestion/service.js:224
        for (const block of blocks) {
                            ^

TypeError: blocks is not iterable
    at L2IngestionService._syncSequencerBlocks (/opt/optimism/packages/data-transport-layer/dist/src/services/l2-ingestion/service.js:224:29)
    at async L2IngestionService._start (/opt/optimism/packages/data-transport-layer/dist/src/services/l2-ingestion/service.js:147:17)
    at async L2IngestionService.start (/opt/optimism/packages/common-ts/dist/base-service/base-service.js:33:9)
    at async Promise.all (index 2)
    at async L1DataTransportService._start (/opt/optimism/packages/data-transport-layer/dist/src/services/main/service.js:69:13)
    at async L1DataTransportService.start (/opt/optimism/packages/common-ts/dist/base-service/base-service.js:33:9)
    at async /opt/optimism/packages/data-transport-layer/dist/src/services/run.js:77:9

To Reproduce Steps to reproduce the behavior:

1. Starting the docker hub image ethereumoptimism/data-transport-layer:0.5.42 also tested with 0.5.41 with the parameters above

Expected behavior No complete crash instead a warning or detailed error message what is wrong.

System Specs:

• OS: See docker image
• Package Version (or commit hash): ethereumoptimism/data-transport-layer:0.5.42

Additional context I am running the container inside of Kubernetes.

[smartcontracts]

Hmmm confirmed. For now I would recommend using a provider like Alchemy, Infura, or QuickNode rather than syncing from the public endpoint (mainnet.optimism.io). Fix is relatively straightforward, surprised this hasn't been reported before.

[cgebe]

As a workaround, I have started a full sync again via L1 (from my node) now, as stated in Running a testnet or mainnet node documentation. Thanks for the quick response!

[smartcontracts]

Alright cool, L1 sync is safer anyway. We may not be able to get #3342 merged any time soon since anything that touches the DTL is considered protocol critical (which means stricter review) and we have other pressing issues to tackle first (related to shipping the new Bedrock upgrade). Whenever we change the DTL we also need to sync a full node based on those changes which can take a while. In the meantime L1 sync is a great option or alternatively sync via some different endpoint like I mentioned previously.

[Aman-Maheshwari7]

Still getting this error while configuring nodes using docker and keeping NETWORK_NAME=goerli and SYNC_SOURCE=l2. Sync source l1 is working fine! But l2 isn't. (Also I have changed DATA_TRANSPORT_LAYER__RPC_ENDPOINT to L2 node when Imy sync source was l2. So this wasn't the problem) Is this a bug? Or can anyone guide me on this?