ldeffenb

Note: I am adding some pullstorage metrics to my node to count the starts and finishes of Cursors() and a logger.debug in handleCursors() to show me their timings. So the...

Here's a Grafana view of the CPU/Connectivity correlation graphs over the past 2 days. https://gateway.ethswarm.org/access/5073c6779df5d2889356730efa4aee92f0b892fa9a148677f924ecdba32d9128

The first timing log on the Cursors() call: ``` time="2021-11-05T10:08:01-04:00" level=debug msg="pullsync:cursorHandler:Cursors() took 3m7.689375318s for 61353f0ba6f661ff8d0b06e64ac8d6e28f7d491752d78bb0f15a36cf66765acb" time="2021-11-05T10:13:48-04:00" level=debug msg="pullsync:cursorHandler:Cursors() took 3m12.577957385s for 562735c0e18c83a47bb6612f63b970673421d91434c6430c4dc7e98c1b651d95" time="2021-11-05T10:14:08-04:00" level=debug msg="pullsync:cursorHandler:Cursors() took 3m16.600494236s for 46790f28e8682d3fc580ba9397a6b206f41f4801f814cb1fb5f16f6301282bcb"...

And for the final correlation between Cursors() activity and CPU consumption for large localstores (regardless of the ignored context, these peers were still actively connected): https://gateway.ethswarm.org/access/90663dde0b22abc2bd2bcf3c0cc964a6da5766f86924bac2cfb93a449c6116e5 This is the pullstorage...

Here's another extreme view of the slow cursors... Yes, that says there were and still are 175+ active goroutines all trying to build Cursors responses. https://gateway.ethswarm.org/access/19f358a6729e89cbe08f98c4a680d97e0c20a525e9d956699a9e6314d8078fbd

And finally, here's the response time (if you call hours a response) of the concurrent Cursors requests. Working on putting a singleFlight around this with a Forget so that it...

Well, I'm glad somebody did the trailblazing to identify that large storage nodes have some issues before the whole world starts loading up lots of data. In the meantime, I've...

Any plans to publish guidance on this? In particular, how to detect if the issue exists within a node so that we don't just start nuking everything and dropping retrievability...

If you do the following command, substituting the proper IP and debug port, what value is displayed? It should be 2 or 3 on testnet and 8 or 9 on...

Are you sure you have inbound connections open and forwarded to your p2p-addr (default 1634)? With a depth of only 6, it seems that you may not be receiving inbound...