Eth1 syncing/TTD polling with Geth in Sepolia using HTTP or `--web3-force-polling=true` stalls

[tersec]

To reproduce on Linux:

1. Ensure one's starting without any data directories present, either Geth or Nimbus.
2. Run Geth: go-ethereum/build/bin/geth --sepolia --datadir ${HOME}/geth_sepolia/ --http --http.api "personal,engine,eth,web3,net,debug" --ws --ws.api "personal,engine,eth,web3,net,debug" --http.corsdomain "*" --ws.origins "*" --authrpc.jwtsecret=/tmp/jwtsecret console
3. Run Nimbus: nimbus-eth2/build/nimbus_beacon_node --network=sepolia --web3-url=http://127.0.0.1:8551 --rest --web3-force-polling=false --log-level=DEBUG --jwt-secret="/tmp/jwtsecret

Nimbus Eth1-topic logs, in their entirety, are:

$ grep -F '"eth1"' nimbus.log 
{"lvl":"INF","ts":"2022-07-12 16:20:37.943+00:00","msg":"Starting Eth1 deposit contract monitoring","topics":"eth1","contract":"0x7f02c3e3c98b133055b8b348b2ac625669ed295d"}
{"lvl":"DBG","ts":"2022-07-12 16:20:37.957+00:00","msg":"Starting Eth1 syncing","topics":"eth1","from":"1273020:491ebac1(deposits = 0)"}
{"lvl":"INF","ts":"2022-07-12 16:20:37.960+00:00","msg":"startEth1Syncing: checking for merge terminal block","topics":"eth1","currentEpoch":4971,"BELLATRIX_FORK_EPOCH":100,"totalDifficulty":"131072","ttd":"17000000000000000","terminalBlockHash":{"val":"0x0000000000000000000000000000000000000000000000000000000000000000","has":false}}

Geth's view of eth.getBlockByNumber("latest"), which Nimbus is using here to detect Eth1 chain progress:

WARN [07-12|16:39:11.425] Local chain is post-merge, waiting for beacon client sync switch-over... 
> eth.getBlockByNumber("latest")
{
  baseFeePerGas: "0x3b9aca00",
  difficulty: "0x20000",
  extraData: "0x5365706f6c69612c20417468656e732c204174746963612c2047726565636521",
  gasLimit: "0x1c9c380",
  gasUsed: "0x0",
  hash: "0x25a5cc106eea7138acab33231d7160d69cb777ee0c2c553fcddf5138993e6dd9",
  logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  miner: "0x0000000000000000000000000000000000000000",
  mixHash: "0x0000000000000000000000000000000000000000000000000000000000000000",
  nonce: "0x0000000000000000",
  number: "0x0",
  parentHash: "0x0000000000000000000000000000000000000000000000000000000000000000",
  receiptsRoot: "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  sha3Uncles: "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
  size: "0x225",
  stateRoot: "0x5eb6e371a698b8d68f665192350ffcecbbbf322916f4b51bd79bb6887da3f494",
  timestamp: "0x6159af19",
  totalDifficulty: "0x20000",
  transactions: [],
  transactionsRoot: "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  uncles: []
}

that is, the genesis block, despite that "Local chain is post-merge, waiting for beacon client sync switch-over."

To get a sense of just how much Geth at this point is just idly looping, waiting for Nimbus to do something, verbatim, no-editing logs:

WARN [07-12|16:41:51.167] Unexpected trienode heal packet          peer=8426e158 reqid=476,292,901,603,602,147
WARN [07-12|16:41:54.843] Unexpected trienode heal packet          peer=02bf54e0 reqid=7,801,068,405,742,975,406
WARN [07-12|16:41:58.928] Local chain is post-merge, waiting for beacon client sync switch-over... 
WARN [07-12|16:41:58.994] Unexpected trienode heal packet          peer=8426e158 reqid=3,686,117,766,686,008,108
WARN [07-12|16:42:02.908] Unexpected trienode heal packet          peer=8426e158 reqid=194,165,526,427,036,032
WARN [07-12|16:42:06.930] Unexpected trienode heal packet          peer=8426e158 reqid=5,462,652,039,868,756,088
WARN [07-12|16:42:10.897] Local chain is post-merge, waiting for beacon client sync switch-over... 
WARN [07-12|16:42:10.979] Unexpected trienode heal packet          peer=8426e158 reqid=922,466,164,104,522,931
WARN [07-12|16:42:14.750] Unexpected trienode heal packet          peer=296efc6c reqid=954,072,685,676,102,586
WARN [07-12|16:42:18.941] Unexpected trienode heal packet          peer=8426e158 reqid=4,563,193,042,630,460,976
WARN [07-12|16:42:22.875] Local chain is post-merge, waiting for beacon client sync switch-over... 
WARN [07-12|16:42:22.948] Unexpected trienode heal packet          peer=8426e158 reqid=84,408,977,617,559,864
WARN [07-12|16:42:27.106] Unexpected trienode heal packet          peer=8426e158 reqid=1,648,193,764,274,892,498
WARN [07-12|16:42:30.836] Unexpected trienode heal packet          peer=81aa411a reqid=2,281,671,756,749,681,025
WARN [07-12|16:42:34.842] Local chain is post-merge, waiting for beacon client sync switch-over... 
WARN [07-12|16:42:34.960] Unexpected trienode heal packet          peer=8426e158 reqid=9,021,745,732,540,736,967
WARN [07-12|16:42:38.666] Unexpected trienode heal packet          peer=62696f86 reqid=9,038,985,427,848,713,050
WARN [07-12|16:42:42.479] Unexpected trienode heal packet          peer=62696f86 reqid=1,898,931,304,352,634,404
WARN [07-12|16:42:46.286] Local chain is post-merge, waiting for beacon client sync switch-over... 
WARN [07-12|16:42:46.295] Unexpected trienode heal packet          peer=296efc6c reqid=6,875,107,594,465,832,364
WARN [07-12|16:42:50.231] Unexpected trienode heal packet          peer=8426e158 reqid=7,843,023,984,466,394,169
WARN [07-12|16:42:53.968] Unexpected trienode heal packet          peer=296efc6c reqid=7,323,970,397,002,370,638
WARN [07-12|16:42:57.772] Local chain is post-merge, waiting for beacon client sync switch-over... 

Geth will not continue until Nimbus starts feeding it newPayload/forkchoiceUpdated calls, and Nimbus will not do so until it thinks the merge has happened. If it solely waited for TTD, this would deadlock.

This hinders both Eth1 syncing and TTD detection on merge networks with Geth using HTTP or --web3-force-polling=true. The latter is an undocumented setting that we don't need to support, per se, but HTTP Web3 URLs imply it.

Geth commit:

commit d740d6e741562e98b3d75a98e7decdfee3bdd904 (HEAD -> master, origin/master, origin/HEAD)
Merge: 48da9d551 68cd0cda4
Author: Péter Szilágyi <[email protected]>
Date:   Tue Jul 12 14:39:05 2022 +0300

    Merge pull request #25290 from karalabe/tar-archive-folder-time
    
    internal/build: add a timestamp to the tar archive folder