ceph.functests.1node tries to install non-existent package, but only when run via CLI

[smithfarm]

Description of Issue/Question

This is an odd issue, but 100% reproducible on master. When run via salt-run state.orch, the ceph.functests.1node orchestration works flawlessly. But when run via CLI, it fails like so:

2018-10-28T02:10:19.406 INFO:teuthology.orchestra.run.target158069065112.stdout:[70/79] Executing state ceph.apparmor.default-enforce...
2018-10-28T02:10:19.406 INFO:teuthology.orchestra.run.target158069065112.stdout:         target158069065112.teuthology... fail
2018-10-28T02:10:19.406 INFO:teuthology.orchestra.run.target158069065112.stdout:Finished stage ceph.functests.1node: succeeded=69/79 failed=1/79
2018-10-28T02:10:19.406 INFO:teuthology.orchestra.run.target158069065112.stdout:
2018-10-28T02:10:19.407 INFO:teuthology.orchestra.run.target158069065112.stdout:Failures summary:
2018-10-28T02:10:19.407 INFO:teuthology.orchestra.run.target158069065112.stdout:ceph.apparmor.default-enforce:
2018-10-28T02:10:19.407 INFO:teuthology.orchestra.run.target158069065112.stdout:  target158069065112.teuthology:
2018-10-28T02:10:19.407 INFO:teuthology.orchestra.run.target158069065112.stdout:    install apparmor: An error was encountered while installing package(s): Zypper command failure: Package 'apparmor' not found.
2018-10-28T02:10:19.408 INFO:teuthology.orchestra.run.target158069065112.stdout:    apparmor: The named service apparmor is not available
2018-10-28T02:10:19.408 INFO:teuthology.orchestra.run.target158069065112.stdout:    /etc/apparmor.d/usr.bin.ceph-mds: Parent directory not present
2018-10-28T02:10:19.408 INFO:teuthology.orchestra.run.target158069065112.stdout:    /etc/apparmor.d/usr.bin.ceph-mgr: Parent directory not present
2018-10-28T02:10:19.409 INFO:teuthology.orchestra.run.target158069065112.stdout:    /etc/apparmor.d/usr.bin.ceph-mon: Parent directory not present
2018-10-28T02:10:19.409 INFO:teuthology.orchestra.run.target158069065112.stdout:    /etc/apparmor.d/usr.bin.ceph-osd: Parent directory not present
2018-10-28T02:10:19.409 INFO:teuthology.orchestra.run.target158069065112.stdout:    /etc/apparmor.d/usr.bin.radosgw: Parent directory not present
2018-10-28T02:10:19.409 INFO:teuthology.orchestra.run.target158069065112.stdout:    aa-enforce /etc/apparmor.d/usr.bin.ceph-mds: Command "aa-enforce /etc/apparmor.d/usr.bin.ceph-mds" run
2018-10-28T02:10:19.410 INFO:teuthology.orchestra.run.target158069065112.stdout:        stdout:
2018-10-28T02:10:19.410 INFO:teuthology.orchestra.run.target158069065112.stdout:        stderr: /bin/sh: aa-enforce: command not found
2018-10-28T02:10:19.410 INFO:teuthology.orchestra.run.target158069065112.stdout:    aa-enforce /etc/apparmor.d/usr.bin.ceph-mgr: Command "aa-enforce /etc/apparmor.d/usr.bin.ceph-mgr" run
2018-10-28T02:10:19.410 INFO:teuthology.orchestra.run.target158069065112.stdout:        stdout:
2018-10-28T02:10:19.411 INFO:teuthology.orchestra.run.target158069065112.stdout:        stderr: /bin/sh: aa-enforce: command not found
2018-10-28T02:10:19.411 INFO:teuthology.orchestra.run.target158069065112.stdout:    aa-enforce /etc/apparmor.d/usr.bin.ceph-mon: Command "aa-enforce /etc/apparmor.d/usr.bin.ceph-mon" run
2018-10-28T02:10:19.411 INFO:teuthology.orchestra.run.target158069065112.stdout:        stdout:
2018-10-28T02:10:19.411 INFO:teuthology.orchestra.run.target158069065112.stdout:        stderr: /bin/sh: aa-enforce: command not found
2018-10-28T02:10:19.412 INFO:teuthology.orchestra.run.target158069065112.stdout:    aa-enforce /etc/apparmor.d/usr.bin.ceph-osd: Command "aa-enforce /etc/apparmor.d/usr.bin.ceph-osd" run
2018-10-28T02:10:19.412 INFO:teuthology.orchestra.run.target158069065112.stdout:        stdout:
2018-10-28T02:10:19.412 INFO:teuthology.orchestra.run.target158069065112.stdout:        stderr: /bin/sh: aa-enforce: command not found
2018-10-28T02:10:19.412 INFO:teuthology.orchestra.run.target158069065112.stdout:    aa-enforce /etc/apparmor.d/usr.bin.radosgw: Command "aa-enforce /etc/apparmor.d/usr.bin.radosgw" run
2018-10-28T02:10:19.413 INFO:teuthology.orchestra.run.target158069065112.stdout:        stdout:
2018-10-28T02:10:19.413 INFO:teuthology.orchestra.run.target158069065112.stdout:        stderr: /bin/sh: aa-enforce: command not found
2018-10-28T02:10:19.413 INFO:teuthology.orchestra.run.target158069065112.stdout:    aa-enforce /etc/apparmor.d/usr.sbin.httpd-prefork: Command "aa-enforce /etc/apparmor.d/usr.sbin.httpd-prefork" run
2018-10-28T02:10:19.413 INFO:teuthology.orchestra.run.target158069065112.stdout:        stdout:
2018-10-28T02:10:19.414 INFO:teuthology.orchestra.run.target158069065112.stdout:        stderr: /bin/sh: aa-enforce: command not found
2018-10-28T02:10:19.414 INFO:teuthology.orchestra.run.target158069065112.stdout:    aa-enforce /etc/apparmor.d/usr.sbin.oaconfig: Command "aa-enforce /etc/apparmor.d/usr.sbin.oaconfig" run
2018-10-28T02:10:19.414 INFO:teuthology.orchestra.run.target158069065112.stdout:        stdout:
2018-10-28T02:10:19.414 INFO:teuthology.orchestra.run.target158069065112.stdout:        stderr: /bin/sh: aa-enforce: command not found
2018-10-28T02:10:19.554 ERROR:tasks.deepsea:WWWW: orchestration ceph.functests.1node failed. Here comes journalctl!
2018-10-28T02:10:19.555 INFO:teuthology.orchestra.run.target158069065112:Running: 'sudo journalctl --all'

As far as I can tell, the os_switch(custom) thingy is failing to recognize the SLE-15 environment, but only when run via the CLI???

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

This is trivial to reproduce, and Jenkins will start reproducing it as soon as https://github.com/SUSE/ceph/pull/223 is merged, because it adds a CLI/non-CLI dimension to the tier1 test matrix.

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Single-node cluster with mds and rgw roles. Run Stages 0-4. Run ceph.functests.1node orchestration via deepsea command (CLI).

Versions Report

(Provided by running: salt-run deepsea.version rpm -qi salt-minion rpm -qi salt-master )

2018-10-28T01:52:39.743 INFO:teuthology.orchestra.run.target158069065112:Running: 'rpm -q salt-master'
2018-10-28T01:52:39.836 INFO:teuthology.orchestra.run.target158069065112.stdout:salt-master-2018.3.0-5.12.1.x86_64
2018-10-28T01:52:39.838 INFO:teuthology.orchestra.run.target158069065112:Running: 'rpm -q salt-minion'
2018-10-28T01:52:39.882 INFO:teuthology.orchestra.run.target158069065112.stdout:salt-minion-2018.3.0-5.12.1.x86_64
2018-10-28T01:52:39.885 INFO:teuthology.orchestra.run.target158069065112:Running: 'rpm -q salt-api'
2018-10-28T01:52:39.991 INFO:teuthology.orchestra.run.target158069065112.stdout:salt-api-2018.3.0-5.12.1.x86_64
2018-10-28T01:52:39.993 INFO:teuthology.orchestra.run.target158069065112:Running: 'type python2 > /dev/null 2>&1'
2018-10-28T01:52:40.033 INFO:tasks.deepsea.preflight:python2 not installed on master node
2018-10-28T01:52:40.034 INFO:teuthology.orchestra.run.target158069065112:Running: 'type python3 > /dev/null 2>&1'
2018-10-28T01:52:40.645 INFO:teuthology.orchestra.run.target158069065112:Running: 'python3 --version'
2018-10-28T01:52:41.005 INFO:teuthology.orchestra.run.target158069065112.stdout:Python 3.6.5
2018-10-28T01:52:41.006 INFO:teuthology.orchestra.run.target158069065112:Running: 'type deepsea > /dev/null 2>&1'
2018-10-28T01:52:41.369 INFO:teuthology.orchestra.run.target158069065112:Running: 'deepsea --version'
2018-10-28T01:52:44.054 INFO:teuthology.orchestra.run.target158069065112.stdout:deepsea 0.9.7+git.0.e3fe25b6a

(Also reproduces on tip of master branch when installed from source.)

[smithfarm]

@rjfd If you need a cloud environment that is reproducing this bug, just let me know...

[smithfarm]

Jenkins will start reproducing it

Jenkins is not reproducing it, because I moved the CLI version of the functional test to tier3. So, to run the test in a PR, write susebot run teuthology deepsea/tier3/functional.

[jschmid1]

@rjfd could you please have a look?

[smithfarm]

@jschmid1 Given that this error only reproduces with the CLI, it's not a blocker for testing. We can still run the functests in non-CLI mode.

[jschmid1]

It's still a blocker for the product :>