featurebase
featurebase copied to clipboard
ignore shards which node does not own
Description
in some situations, e.g. after a resize, a node may have shard files which do not belong to it. Normally the node will open all files in its data directory regardless of whether it needs to use them this can lead to an explosion of used file descriptors.
Success criteria (What criteria will consider this ticket closeable?
Pilosa does not open data files which it does not need
Had to back this change out—seems there is a race between when the cluster knows about all the nodes and when holder opening occurs which was causing #1948
I think the solution is to use cluster.Topology.Nodes
instead of cluster.nodes
, but validating and properly testing that is a bit more than I'm willing to dig into right now.
A resize does actually delete the leftover shards once it's complete, so this work is more important for the case of running multiple Pilosa nodes on the same shared storage medium which isn't a priority at the moment.