chef-server-ctl cleanse --with-external deletes private-chef-secrets.json before cleaning up the database, hilarity ensues

[irvingpop]

I think we may have an order-of-operations problem here: https://github.com/chef/omnibus-ctl/blob/master/lib/omnibus-ctl.rb#L861 - the external_cleanse_X methods shouldn't fire after cleanse because the secrets files will be gone by then.

[root@ip-10-42-49-198 ~]# ls -la /etc/opscode
total 84
drwxr-xr-x  3 root    root  4096 Jul 24 21:19 .
drwxr-xr-x 80 root    root  4096 Jul 24 21:16 ..
-rw-r-----  1 root    root  1452 Jul 24 21:14 chef-server.rb
-rw-r--r--  1 root    root     0 Jul 24 21:14 chef-server.rb.bak
-rw-------  1 opscode root 22636 Jul 24 21:19 chef-server-running.json
-rw-r--r--  1 opscode root   382 Jul 24 21:16 dark_launch_features.json
-rw-r--r--  1 root    root    63 Jul 24 21:19 logrotate.conf
drwxr-xr-x  2 root    root  4096 Jul 24 21:19 logrotate.d
-rw-------  1 opscode root  1679 Jul 24 21:16 pivotal.pem
-rw-r--r--  1 root    root   636 Jul 24 21:19 pivotal.rb
-rw-------  1 root    root 13699 Jul 24 21:19 private-chef-secrets.json
-rw-r--r--  1 root    root   305 Jul 24 21:19 private-chef.sh
-rw-------  1 opscode root  1675 Jul 24 21:16 webui_priv.pem
-rw-r--r--  1 root    root   451 Jul 24 21:16 webui_pub.pem
[root@ip-10-42-49-198 ~]# chef-server-ctl cleanse --with-external
    *******************************************************************
    * * * * * * * * * * *       STOP AND READ       * * * * * * * * * *
    *******************************************************************
    This command will delete *all* local configuration, log, and
    variable data associated with Chef Server.
    This will also delete externally hosted Chef Server data.
    This means that any service you have configured as 'external'
    will have any Chef Server permanently deleted.

    You have 60 seconds to hit CTRL-C before configuration,
    logs, local, and remote data for this application are permanently
    deleted.
    *******************************************************************

ok: down: bookshelf: 0s, normally up
ok: down: nginx: 0s, normally up
ok: down: oc_bifrost: 1s, normally up
ok: down: oc_id: 0s, normally up
ok: down: opscode-chef-mover: 2563s, normally up
ok: down: opscode-erchef: 1s, normally up
ok: down: redis_lb: 0s, normally up
bookshelf disabled, not stopping
nginx disabled, not stopping
oc_bifrost disabled, not stopping
oc_id disabled, not stopping
opscode-chef-mover disabled, not stopping
opscode-erchef disabled, not stopping
redis_lb disabled, not stopping
Terminating processes running under application users. This will take a few seconds.
Your config files have been backed up to /root/opscode-cleanse-2017-07-24T21:59.
Deleting data from external service: opscode-solr4
Cleansing data in a remote Sol4 instance is not currently supported.
Deleting data from external service: postgresql
CLEANSE001: While local cleanse of Chef Server succeeded, an error
            occurred while deleting Chef Server data from the external
            PostgreSQL server at idhz110eswaam1.csenmsxeypge.us-west-2.rds.amazonaws.com.

            The error reported was:

                /etc/opscode/private-chef-secrets.json does not exist

            To complete cleanup of PostgreSQL, please log into PostgreSQL
            on idhz110eswaam1.csenmsxeypge.us-west-2.rds.amazonaws.com as superuser and execute the statements
            that have been saved to the file below:

              /root/2017-07-24T21:59-chef-server-manual-postgresql-cleanup.sql

            See https://docs.chef.io/error_messages.html#cleanse001-postgres-failed
            for more information.

[thompsm]

I've just hit exactly the same problem. Is there any workaround for this until it gets fixed? (Apart from running the sql commands manually).

[irvingpop]

Not as of yet. I think we'll need to have a talk with @chef/chef-server-maintainers about the best way to reorder the operations. Perhaps moving that from cleanse_post_hook to fire before or during cleanup_procs_and_nuke ?

[stevendanna]

I have a feeling that we might broke this with the chef_secrets work. Previously the db password would be in the running config which is read at the start of the command. Now it is in the secrets file which is read lazily as it is needed. We should make sure we read the secrets we need from the start.