enforce upper limit on number of kdcluster members

[joel-bluedata]

We need to take into account that there is an upper limit on the CR document size. When KD modifies the document to write status it could violate that limit.

Even if there wasn't an upper limit enforced by K8s we shouldn't allow the document to get arbitrarily large anyway. I think we can calculate an upper bound on the space taken up by each per-member status, so for a given incoming new or changed document our validator should be able to calculate if the resulting status would be too large and reject the document.

One thing I'm not 100% sure about is whether the observed document size really corresponds to the etcd storage size, for comparing against that limit. Also the limit reported by the API server for body size (3145728 bytes) may be different from the limit etcd has on storage size.

If we can't or don't want to get into fancy calculations, we could just pick a conservative limit for now and hardcode it, like declare "no more than 500 members". That would be a good nearterm solution, especially since we haven't tested kdclusters at absurdly large scales yet.

[joel-bluedata]

We have the simple limit (1000) for 0.5.0. Leaving this issue open to consider any more complex solution.