featurebase icon indicating copy to clipboard operation
featurebase copied to clipboard

ignore shards which node does not own

Open jaffee opened this issue 6 years ago • 1 comments

Description

in some situations, e.g. after a resize, a node may have shard files which do not belong to it. Normally the node will open all files in its data directory regardless of whether it needs to use them this can lead to an explosion of used file descriptors.

Success criteria (What criteria will consider this ticket closeable?

Pilosa does not open data files which it does not need

jaffee avatar Dec 31 '18 19:12 jaffee

Had to back this change out—seems there is a race between when the cluster knows about all the nodes and when holder opening occurs which was causing #1948

I think the solution is to use cluster.Topology.Nodes instead of cluster.nodes, but validating and properly testing that is a bit more than I'm willing to dig into right now.

A resize does actually delete the leftover shards once it's complete, so this work is more important for the case of running multiple Pilosa nodes on the same shared storage medium which isn't a priority at the moment.

jaffee avatar Apr 23 '19 16:04 jaffee