document import existing cluster feature, and kinks thereof

[tserong]

Need to document salt-run populate.engulf_existing_cluster on the wiki. See also #444 and #445.

Additionally, when running stage 4 you might hit "Error EINVAL: entity mds.$HOSTNAME exists but cap mds does not match", because ceph-deploy sets caps mds = "allow", whereas deepsea sets caps mds = "allow *" when deploying MDS.

In general, if you hit key cap mismatches, the "easy" fix seems to be to run ceph auth caps $ENTITY [...caps...] to update the existing key to match what DeepSea is trying to deploy. That may or may not be the most correct fix though (should one instead manually tweak the relevant DeepSea pieces to match the running cluster instead?)

[tserong]

Ping @jan--f, in case you have opinions on keys

[jan--f]

Ok so this sounds like a bug to me. The mds should not have capabilities to change its own settings. We want to limit this to the admin keyring(s).

[jschmid1]

Is that still valid?

[jschmid1]

@jan--f @tserong

[tserong]

Looks like we never fixed #445, so the rgw key permissions are still too loose for the mon (see Florian's comment on that bug for details), and the mds caps still seem to be "allow *", but should really be "allow". We should fix those things, but doing so is going to be annoying for any existing user, because if we change the key caps DS ships, then the user runs stage 2 (which updates the key cache files), then runs stage 3 or 4, this will fail with "entity $foo exists but cap $bar does not match", for all the keys whose caps have changed, and the user will be stuck manually updating all the key caps.

I wonder if we should just make DeepSea force whatever key caps we ship? That would make fixing this not a PITA, and would also make several people happy who have previously complained about needing to manually tweak key caps after an engulf. I had previously taken a conservative view on this (i.e. it's better to fail loudly and not silently change permission type things, so that the user knows what's going on), but I wonder if I was maybe being too paranoid...?

As for this issue (i.e. documenting the engulf procedure on the wiki), I'm somewhat inclined to not bother -- it's already covered quite well in the SES docs.