Performance issue getting eth1DataAndDeposits for block production

[twoeths]

Describe the bug

It takes up to 5.84s to get eth1DataAndDeposit for block production for a mainnet node

Expected behavior

this issue rarely happen, actually it's the 1st time I see it takes >1s, most of the time it should take <=50ms

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.14.0

[nazarhussain]

actually it's the 1st time I see it takes >1s

@tuyennhv Should we not wait additional occurrence with such hype to mark it a bug? If it's just once ever, there could not some non-code related issue.

[twoeths]

Should we not wait additional occurrence with such hype to mark it a bug? If it's just once ever, there could not some non-code related issue.

I believe it's a code issue, we should try to reproduce it in a benchmark test first instead of waiting for new occurrences

[philknows]

Can you confirm @tuyennhv that we're no longer seeing this? I took a look at some of our mainnet Lido nodes and it doesn't seem to be that bad on v1.16

[twoeths]

haven't seen this issue for a while, closing now. Will reopen if we see it again