Lighthouse VC to Lodestar BN time discrepancy errors

[nflaig]

Describe the bug

Lighthouse VC reports time discrepancy between VC and BN. This started to happen when upgrading Lighthouse to v4.2.0.

May 25 14:55:59.013 ERRO Time discrepancy with beacon node       endpoint: http://lodestar:9596/, local_slot: 6516878, beacon_node_slot: 6516876, msg: check the system time on this host and the beacon node
May 25 15:08:47.013 ERRO Time discrepancy with beacon node       endpoint: http://lodestar:9596/, local_slot: 6516942, beacon_node_slot: 6516940, msg: check the system time on this host and the beacon node
May 25 15:32:23.011 ERRO Time discrepancy with beacon node       endpoint: http://lodestar:9596/, local_slot: 6517060, beacon_node_slot: 6517058, msg: check the system time on this host and the beacon node
May 25 15:49:11.012 ERRO Time discrepancy with beacon node       endpoint: http://lodestar:9596/, local_slot: 6517144, beacon_node_slot: 6517142, msg: check the system time on this host and the beacon node
May 25 16:10:11.015 ERRO Time discrepancy with beacon node       endpoint: http://lodestar:9596/, local_slot: 6517249, beacon_node_slot: 6517247, msg: check the system time on this host and the beacon node

See https://github.com/sigp/lighthouse/issues/3421#issuecomment-1764281107 for details on why this happens.

Expected behavior

There should not be any errors about time discrepancy.

Steps to reproduce

1. Run Lodestar BN
2. Attach Lighthouse VC with active validators (might also happen without)
3. Observe Lighthouse VC logs

Additional context

Also reported on discord

• https://discord.com/channels/593655374469660673/593655641445367808/1111151549851385866
• https://discord.com/channels/593655374469660673/593655641445367808/1111325957169623133

See getSyncingStatus beacon API.

Versions

Lodestar: v1.8.0 / v1.9.0-rc.0 Lighthouse: v4.2.0 (does not happen with v4.1.0)

[nflaig]

Related issue

• https://github.com/sigp/lighthouse/issues/3421