Light client missing finalized checkpoint snapshot data on kiln testnet

[hujw77]

Describe the bug

curl -X GET https://lodestar-kiln.chainsafe.io/eth/v1/beacon/headers/651583 | jq .data.root
0x1d7c3b1e5a07e1d68fee047759d2ce5a3d6a618fe467a2fbada3f17dc7e34197

curl -X GET https://lodestar-kiln.chainsafe.io/eth/v1/beacon/headers/651585 | jq .data.root
0x2ed5461e48e1edb4e0bae810017d20d888bf11e5d5b6d60c48ee48050181b3c0

slot 651583 root: 0x1d7c3b1e5a07e1d68fee047759d2ce5a3d6a618fe467a2fbada3f17dc7e34197
slot 651584 missing
slot 651585 root: 0x2ed5461e48e1edb4e0bae810017d20d888bf11e5d5b6d60c48ee48050181b3c0

On kiln testnet, beacause of the slot 651584 is missing, the checkpoint of epoch 20362 is slot 651583, but I get the snapshot in slot 651583 of finalized epoch 20362 using light client api :

curl -X GET https://lodestar-kiln.chainsafe.io/eth/v1/lightclient/snapshot/0x1d7c3b1e5a07e1d68fee047759d2ce5a3d6a618fe467a2fbada3f17dc7e34197 | jq

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "syncCommitteeWitness not available 0x1d7c3b1e5a07e1d68fee047759d2ce5a3d6a618fe467a2fbada3f17dc7e34197"
}

And I get the snapshot at slot 651585, it returns the snapshot.

curl -X GET https://lodestar-kiln.chainsafe.io/eth/v1/lightclient/snapshot/0x2ed5461e48e1edb4e0bae810017d20d888bf11e5d5b6d60c48ee48050181b3c0 | jq

{
  "data": ...
}

Expected behavior Why did the light client store the snapshot at slot 651585 instead of the slot 651583 at epoch 20362 ?

Desktop (please complete the following information):

• OS: ubuntu
• Version: v0.37.0
• Branch:
• Commit hash:

[dapplion]

Thanks for reporting @hujw77 will look into it asap

[philknows]

Hey @hujw77 , we will look into this for you. However, we will be deprecating this endpoint and recommend moving to Ropsten for your needs.

[dadepo]

I was able to confirm this behaviour on Goerli for slot 119920, which has the block for its checkpoint slot 3837440 missing. And requesting the light client bootstrap object using object hash of 3837440 and 3837439 fails but 3837441 works

[dapplion]

Closing since it's related to an old incident. Re-open with details of newer incident if it happens again