extremely slow leveldb iteration during cache eviction

[istae]

The leveldb iterator in the the cache.RemoveOldest function is extremely slow.

To remove one single entry, close to half a second is used. To remove a 1000 entries, close to 45 seconds is used.

As a result, some gateway nodes are struggling to fall under the allowed 1 million chunk capacity.

[istae]

Attached is the cpu profile of bee 2 pprof.bee.samples.cpu.071.pb.gz

[istae]

[istae]

one clue is localstore size may affect things as well.