State by slot not being fetched correctly

[KaloyanTanev]

Describe the bug

Starting attempts to fetch chain state by slot fails before chain has synced to the said slot (expected) but also after (unexpected).

Expected behavior

The BN returns the state at slot after successful sync.

Steps to reproduce

1. Request chain state for a slot (i.e. at POST /eth/v1/beacon/states/{state_id}/validators) couple of seconds before the chain is synced to the slot.
2. Continue retrying each 0.5-1 seconds.
3. The chain syncs to this slot after couple of seconds.
4. The responses to the request are still unknown state.

Additional context

I have reproduced the issue on local Kurtosis devnet, running Charon DV. The setup I have is geth v1.15.5, lodestar CL v1.28.1, Charon v1.3.0, lodestar VC v1.28.1. I have added more logging to v1.28.1 tag and ran lodestar CL from this branch https://github.com/KaloyanTanev/lodestar/pull/1/files. I attach logs from the run.

lodestar.log

Operating system

macOS

Lodestar version or commit hash

v1.28.1

[Sandijigs]

I would love to work on this issue , Please assign it to me

[philknows]

Please pick something and continue working on it before being assigned others @Sandijigs