go-ethereum icon indicating copy to clipboard operation
go-ethereum copied to clipboard

How does geth communicate with prysm?

Open lcgogo opened this issue 3 years ago • 17 comments

Now I run geth 1.10.23 and prysm 3.0.0 with prysm connected to geth 8551 port by a jwt.hex file

But in geth 1.10.23 logs always print WARN like below

WARN [08-24|13:51:42.419] Merge is configured, but no beacon client seen. Please ensure you have one available before the transition arrives! eta=3w1d28m

lcgogo avatar Aug 24 '22 13:08 lcgogo

Geth usually waits a bit for the consensus client to attach and start sending some exchanges. If it does not, it will complain. Let it run for a while and see if the complaints remain. According to the spec the consensus client should ping us every minute at max for a ttd config exchange. We first wait 30 secs, then 5 minutes after for this exchange and keep warning until we see it,

karalabe avatar Aug 24 '22 14:08 karalabe

Teku seems to be waiting for the Bellatrix fork (epoch 144896 – 11:34:47am UTC on Sept 6, 2022) to monitor EL for TTD. Maybe Prysm does the same?

Neurone avatar Aug 24 '22 23:08 Neurone

I checked on Prysm source code, and it seems it also waits for the Bellatrix fork before starting to monitor EL.

// Checks the transition configuration between Prysm and the connected execution node to ensure
// there are no differences in terminal block difficulty and block hash.
// If there are any discrepancies, we must log errors to ensure users can resolve
//the problem and be ready for the merge transition.
func (s *Service) checkTransitionConfiguration(
	ctx context.Context, blockNotifications chan *feed.Event,
) {
	// If Bellatrix fork epoch is not set, we do not run this check.
	if params.BeaconConfig().BellatrixForkEpoch == math.MaxUint64 {
		return
	}
...

Neurone avatar Aug 25 '22 00:08 Neurone

Although, it should have been set that variable few days ago:

mainnetBellatrixForkEpoch = 144896 // Sept 6, 2022, 11:34:47am UTC
...
BellatrixForkEpoch:   mainnetBellatrixForkEpoch,

So, maybe that is not the issue.

Neurone avatar Aug 25 '22 00:08 Neurone

And yes, it waits too:

if currentEpoch >= params.BeaconConfig().BellatrixForkEpoch && !hasTtdReached {
	hasTtdReached, err = s.logTtdStatus(ctx, ttd)
	if err != nil {
		log.WithError(err).Error("Could not log ttd status")
	}
}

Neurone avatar Aug 25 '22 00:08 Neurone

@Neurone Many thanks. I see the printout every 5 mins. Should I wait prysm full synced? I run geth and prysm in diff docker containers. Is there any method to test connection from geth side manually?

Geth usually waits a bit for the consensus client to attach and start sending some exchanges. If it does not, it will complain. Let it run for a while and see if the complaints remain. According to the spec the consensus client should ping us every minute at max for a ttd config exchange. We first wait 30 secs, then 5 minutes after for this exchange and keep warning until we see it,

lcgogo avatar Aug 25 '22 02:08 lcgogo

on your prysm beacon, you should see that it has connected to the CL like this

Aug 25 03:36:06 hqprdubeth start_mainnet_beacon.sh[3777890]: time="2022-08-25 03:36:06" level=info msg="Connected to new endpoint: http://localhost:8550" prefix=powchain

garyng2000 avatar Aug 25 '22 03:08 garyng2000

on your prysm beacon, you should see that it has connected to the CL like this

Aug 25 03:36:06 hqprdubeth start_mainnet_beacon.sh[3777890]: time="2022-08-25 03:36:06" level=info msg="Connected to new endpoint: http://localhost:8550" prefix=powchain

Understood. Many thanks.

lcgogo avatar Aug 25 '22 05:08 lcgogo

So just to clarify, getting this message is expected until the Bellatrix fork or not?

WARN [08-24|13:51:42.419] Merge is configured, but no beacon client seen. Please ensure you have one available before the transition arrives! eta=3w1d28m

ailiao avatar Aug 26 '22 07:08 ailiao

So just to clarify, getting this message is expected until the Bellatrix fork or not?

WARN [08-24|13:51:42.419] Merge is configured, but no beacon client seen. Please ensure you have one available before the transition arrives! eta=3w1d28m

After correctly config beacon node, this message disappeared. Now I use below option instead of http+port in prysm beacon node according to https://docs.prylabs.network/docs/install/install-with-docker

  --execution-endpoint=/path/to/geth.ipc

lcgogo avatar Aug 26 '22 07:08 lcgogo

Nimbus client behaves the same way and the message is supposed to disappear after Bellatrix fork as well.

A post from dev on Discord:

For users running the latest execution clients, even though you've configured the authenticated port (using the JWT secret options) you may see a warnings like the following:

WARN [08-23|09:21:08.395] Merge is configured, but no beacon client seen. Please ensure you have one available before the transition arrives! (geth) WARN | EngineExchangeTransitionConfiguration | not called in 120 seconds, consensus client may not be connected (besu)

This is normal - Nimbus (v22.8.0) will activate the merge mode as it enters the Bellatrix hard fork on September 6, and the warning will go away. Until then, keep on eye on nimbus logs instead - if there is anything wrong with the connection to geth, a WRN log will appear on the Nimbus side.

benesjan avatar Aug 30 '22 06:08 benesjan

Yeah you're geth client was not registering the prysm. Question i have is if i am trying to sync geth do i need the conensus layer to finish first before geth is able to process from block 0?

FahdW avatar Sep 02 '22 03:09 FahdW

Bellatrix is done but this damned written still appears... any suggestion about that? P.s. i am using Geth+Nimbus and my connection is through IPC (local socket)

cortexyphan avatar Sep 06 '22 12:09 cortexyphan

@cortexyphan had similar behaviour. After bump nimbus version to latest (22.9.0) and restart both (geth+nimbus), message disappeared in geth logs. Try this out and wait for sync nimbus node (in my case was since "slot=4655914 epoch=145497 sync="03d09h48m (62.57%)")

MindPhaser34 avatar Sep 09 '22 04:09 MindPhaser34

We are also getting a similar error, we are on geth 1.10.23 and beacon chain is on prysm 3.1.1 -- Warning remains as below Merge is configured, but previously seen beacon client is offline. Please ensure it is operational before the transition arrives!

Do we need to update anything or we wait till transition occurs on 14th ? No other errors on the logs. @karalabe

Sreeram1993 avatar Sep 12 '22 13:09 Sreeram1993

We are also getting a similar error, we are on geth 1.10.23 and beacon chain is on prysm 3.1.1

this message is sent in a loop (every 5 seconds), and it will be sent infinitely until the beacon client connects to Geth. After the beacon client connects to Geth it will continuously call ExchangeTransitionConfigurationV1 method via Geth's RPC and this is is how Geth will know that the connection is established and the node is "merge is ready" now, so the message will stop appearing in the log. Check the last date of this message, if it is older than 5 seconds (comparing to current date), then you have merge-ready node.

@Sreeram1993

nuliknol avatar Sep 12 '22 20:09 nuliknol

Yeah you're geth client was not registering the prysm. Question i have is if i am trying to sync geth do i need the conensus layer to finish first before geth is able to process from block 0?

same problem!

LyyDandelion avatar Sep 19 '22 10:09 LyyDandelion

Communication issues between CL and geth should be fixed by now, this issue has seen no activity for the past three months. If you are still having problems with CL-EL comms, please file a new ticket with version-info and logs.

holiman avatar Dec 20 '22 14:12 holiman