Continue to run test suite even if one test fails

[mike-nguyen]

Problem

Ansible will fail if any task fails by default. This is great if using Ansible as a configuration tool, but not so great if using Ansible as a test framework. In Ansible 2.0, the block/rescue/always concept was introduced--allowing us to catch errors and continue to do other things. So why not do that with a set of tests?

Benefits Over Current Approach

• By catching errors and continuing to run tests, users are able to get more test results and could find more bugs sooner. This is especially true when there is a bug that is being fixed but not yet put into a compose.
• All done natively in Ansible

Proposal

1. Separate tests into individual yml files
2. Create a role that will import the above yml files in a block/rescue that will save the test status in a dictionary.
3. Create an entry point playbook, that calls the above role for each of the tests and pass/fail based on whats in the dictionary.

See simple example here: https://github.com/projectatomic/atomic-host-tests/compare/master...mike-nguyen:tester_poc

Potential Issues

• Might not work with the way improved-sanity-test is laid out
• Compatibility with Standard Test Interface https://fedoraproject.org/wiki/CI/Standard_Test_Interface
• Tagging tests
• Triaging failures might be more difficult if prior test leaves system in undesired state

[miabbott]

Could you also add a section for the benefits to this approach?

Being able to be more resilient to failures would greatly improve our ability to run tests, so I'm in favor of that portion.

To the point of the improved-sanity-test, we can always change how that test is laid out. Maybe have a single sanity.yml that does multiple import_playbook? Alternatively, just make the sanity test one giant playbook instead of splitting it up into three playbooks in the file itself.