Ideas for regression testing using ansible

[ben-moss]

In testing a recent PR, I used some ansible ad-hoc commands to check some of the desired outcomes of the PR. This idea can be extended to build in some regression testing of certain aspects of system, so that potential issues with assumptions are caught earlier in the process/pipeline.

Here's just one example:

Proof of corosync availability from on the RTI:

$ ANSIBLE_HOST_KEY_CHECKING=False ansible hanadbnodes -i hosts -a 'zypper se "corosync"' --become | grep corosync
[WARNING]: Consider using the zypper module rather than running 'zypper'.  If you need to use command because zypper is
insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this
message.
[WARNING]: Platform linux on host 10.1.1.5 is using the discovered Python interpreter at /usr/bin/python, but future installation
of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.8/reference_appendices/interpreter_discovery.html for more information.
i | corosync          | The Corosync Cluster Engine and Application Programming Interfaces | package
  | libcorosync-devel | The Corosync Cluster Engine Development Kit                        | package
i | libcorosync4      | The Corosync Cluster Engine Libraries                              | package
[WARNING]: Platform linux on host 10.1.1.4 is using the discovered Python interpreter at /usr/bin/python, but future installation
of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.8/reference_appendices/interpreter_discovery.html for more information.
i | corosync          | The Corosync Cluster Engine and Application Programming Interfaces | package
  | libcorosync-devel | The Corosync Cluster Engine Development Kit                        | package
i | libcorosync4      | The Corosync Cluster Engine Libraries                              | package

Originally posted by @ben-moss in https://github.com/Azure/sap-hana/issues/337#issuecomment-603806381

## Potential Tasks

• [ ] Identify some valuable use cases (e.g. where issues have occurred in the past, and troubleshooting time is costly)
• [ ] Build some simple mechanics to hook into the pipeline/deployment
• [ ] Consider making these types of checks part of standard PR checklist/practice

[rukawata]

Currently our pipeline separates terraform and ansible. I see this a task(s) that checks the result of both:

1. using ansible to check any issues with terraform deployment
2. using ansible to check any issues with ansible configuration

This can be achieved:

1. as part of the current pipeline, using command line, add post-terraform-check and post-ansible-check.
2. or to have them as part of the ansible playbook (eg. add task pre-check, post-check to the current sap_playbook.yml). Customer can choose to turn off the pre/post check if they don't need that, but they would also have the ability to validate their deployment.

For the test case, I've seem below problem that caused trouble in the past:

1. if the repo is configured correctly.
2. hostname not setup properly by waagent.

[ben-moss]

Rather than add to the pipeline, I would advocate adding into the deployment itself. This way the quality stays with the system, and doesn't only run in the context of CI.