[Architecture] ESXi [and probably other] modules don't follow salt/CM best practice

[dfidler]

salt's biggest market value lies in the fact that it can do declarative config management across a wide variety of IT asset types.

A fundamental requirement to declarative configuration management, is that managed objects [henceforth, asset] MUST have an identity in the management system. However, when you look at how the esxi sddc module works, it deprives salt of the identity of the esxi hosts that you're managing.

The modules are designed to be executed on a minion that has the vmware_config pillar data attached to it. When you want to run a command (like set_advanced_config), you specify the "host_name[s]" in the method call to filter which hosts get the config parameter.

The invocation looks like this:

salt saltmaster vmware_esxi.set_advanced_config Power.PerfBias 18

This will set the config for all hosts. Optionally, you can supply a "host_name[s]" parameter to filter which hosts get the config. If the esxi state module had a "set_advanced_config option" (it doesn't yet), I would expect it to look like this:

# /srv/salt/esxi/power_perf_bias.sls
ensure power perf bias set:
  vmware_esxi.advanced_config_present:
    - name: Power.PerfBias
    - value: 18
    - hosts:
      - esxihost01
      - esxihost02
      - etc

What this is doing is forcing you to define your infrastructure inside of your state files. You could abstract this away using pillar and jinja loops, etc, but at the end of the day, the only "identity" that the vsphere environment has is behind a single salt-minion; like this:

salt saltmaster state.apply esxi.power_perf_bias

All changes would be aggregated into one big changes block.

The "salty" way to do this would be to have every host have a minion_id (esxihost01, esxihost02, etc) and then you'd have the following state file:

# /srv/salt/esxi/power_perf_bias.sls
ensure power perf bias set:
  vmware_esxi.advanced_config_present:
    - name: Power.PerfBias
    - value: 18

And you'd configure this in your top.sls

base:
  'esxihost*':
    - esxi.power_perf_bias

Now you can execute highstate against individual systems and easily see which objects had changes, etc.

Why is this important?

Primarily for SecOps. I keep getting requests from customers for them to be able to "assure my esxi configs using my standard benchmarks - I want to ensure that all of my esxi servers in a given cluster have the same configuration and if one changes, I want to know about it".

Customers want the SecOps reports, and events to show that data. I have received "standard operating environment" specifications from a couple of customers now and they include configurations at several levels - the host, vsphere, datacenter, cluster and vm levels.

Using the modules the way that they are currently architected means that any deviation from a standard config shows up as a deviation of the entire environment instead of just for a specific object within it. There's not enough granularity.

I understand why the esxi module was written the way it was; it more closely mirrors the vsphere api, it removes the necessity to stand up hundreds of salt-proxy processes (one for each host in your environment). This method is also loosly akin to managing infrastructure objects inside of aws (using the boto modules).

Older efforts to manage esxi were using a proxy minion (in lieu of a native agent) to give the different management objects identity inside of salt. We have several of them:

• ESXi Host: https://docs.saltproject.io/en/latest/ref/proxy/all/salt.proxy.esxi.html#module-salt.proxy.esxi (useful because it allows power CLI commands to the host and gives the host identity - this seems to have been duplicated into this saltext project)
• vCenter: https://docs.saltproject.io/en/latest/ref/proxy/all/salt.proxy.vcenter.html#module-salt.proxy.vcenter (only useful for reporting)
• ESXi Cluster: https://docs.saltproject.io/en/latest/ref/proxy/all/salt.proxy.esxcluster.html#module-salt.proxy.esxcluster (only useful for reporting)
• ESXi Datacenter: https://docs.saltproject.io/en/latest/ref/proxy/all/salt.proxy.esxdatacenter.html#module-salt.proxy.esxdatacenter (only useful for reporting)

This architecture actually makes sense because if you want to configure your vsphere estate, you will do it at different levels; at the host, vecenter, cluster and DC levels. Each of those objects should have identity. There are many users out there that have standard configs that their systems must adhere to and they look to the secops comply module to implement them (because of its reporting and remediation capabilities).

IMHO, this is a "better" (though not ideal) way of managing vsphere. The traditional way would be to stand up these proxy minions (one for each host) and manage the config using highstate and rules in the top.sls.

Aside: Using the current modules, you can work around this by installing a separate salt-minion for each esxi host (each with a config in a different directory) and then assign the vmware_config to each of them. Then, in the state file, reference the "id" grain to limit operations like the "host_name(s)" parameters. Although, IMHO, the idea would be to have a multiplexing super-proxy minion (kind of like the delta proxy) that detects which hosts are in a vsphere environment, and then assumes the identity of all of them but appears, to the salt-master, to be each of them.