ceph-salt
ceph-salt copied to clipboard
ceph
Remove check for admin_minion in cephadm_minions
Currently, all minions with the 'admin' role are also required to have the 'cephadm' role. This means you can't have a node which has the ceph.conf file and admin keyring, which isn't also managed by the orchestrator. Can we change this? Because I want my salt master to have the 'admin' role (so I can run ceph commands from it), but I don't want the salt master to end up running ceph services (so it needs to not have the 'cephadm' role).
If we make this change, we probably also need to add a separate check to ensure bootstrap_minion has the cephadm role. That's really the only requirement, right? The bootstrap minion must have both admin and cephadm roles. Any other node you want to run cluster services must have the cephadm role, and any node you want to be able to run ceph commands on must have the admin role, but the two are otherwise independent.
The only issue I see with this change is that your ceph.conf will be outdated when you move monitors.
Note that we have a cephadm issue [1] to automatically update /etc/ceph/ceph.conf on all hosts, but if your admin minion is not managed by cephadm, then your ceph.conf will not be updated by cephadm.
Is this a problem? Perhaps ceph-salt should only distribute/copy the keyring file, not the ceph.conf. Or we need an explicit ceph-salt command to update /etc/ceph/ceph.conf on all admin minions (e.g. ceph-salt sync).
[1] https://tracker.ceph.com/issues/45378
Dammit, I'd forgotten about that :-/
I guess this is really a variant on #217. The underlying issue is that I want to be able to have some node that will never host ceph services (so ceph orch apply mon won't randomly start a mon, or whatever), but that can still be an admin node.
I'm not sure what a good solution is here..
ceph orch drain <host> add a flag to disable the scheduling for a specific host. See https://github.com/ceph/ceph/pull/34617
maybe we need to make this somehow explicit?
Once that goes in, we could possibly revert the cephadm role (sorry!) then just rely on the user setting ceph orch drain on any host they don't want services scheduled on... Although using the term "drain" for this purpose, when your intent is to exclude a node from the outset strikes me as a bit odd :-)
What about something like:
if minion has role 'admin' or 'cephadm':
- then do "ceph orch host add $minion"
if minion has role 'admin' and not has role 'cephadm':
- then do "ceph orch drain $minion"
Although using the term "drain" for this purpose, when your intent is to exclude a node from the outset strikes me as a bit odd :-)
That might be the reason that Kubernetes uses three terms for this: "drain" for initiating a thing called "eviction". Plus a node key called "NoSchedule"
What about something like:
if minion has role 'admin' or 'cephadm': - then do "ceph orch host add $minion" if minion has role 'admin' and not has role 'cephadm': - then do "ceph orch drain $minion"
LGTM!
The "drain" PR was closed https://github.com/ceph/ceph/pull/34617 so we need to find a different solution
right now, there is no way to mark a host as "unschedulable" for new daemons. That might be enough for. Like a drain-light feature