soup should verify global.sls file can be rendered before starting install

[steelerguy]

I updated from 2.3.61 to 2.3.90 but it was a very painful process. Someone had put in a threshold rule into /opt/so/saltstack/local/pillar/global.sls that had a tab in it instead of just spaces. The install failed when a salt state failed to apply during the install, although very little of the install had happened it was enough that the manager was in a bad state, but soup reported that the install was done and updated the version in /etc/soversion and the global.sls file.

Recovering from this took a couple hours of poking through /usr/sbin/soup, running various parts of the script by hand, commenting out certain steps (like stopping/removing the now non-existant so-dockerregistry image and checking soup version), having to put a sleep in the script so I could remove duplicate entries that would get put into the global.sls in the index_settings section for so-redis, so-kibana, so-logstash, and so-elasticsearch, etc. I was finally to get the manager to update and work again.

The person who added the threshold entry should have realized their mistake when the alert was not suppressed or errors in the logs. I myself usually run so-idstools-restart which also errors out if you make a mistake. I supposed I should have looked at the logs for errors before running soup, but a simple check to make sure the file can render fully, not just the pillars.

Very easy to reproduce, just put a bad character in a threshold entry in /opt/so/saltstack/local/pillar/global.sls and run soup.

Note: This is an airgap installation, but I don't think that matters.