A playbook ending with an include_role failure marked as successfully completed

[dmsimard]

What is the issue ?

I've bumped into this by running this playbook and the output is short enough to include here (but it's also recorded on the demo):

> ansible-playbook -i hosts with_nginx.yaml

PLAY [Deploy and test ARA API with nginx, authentication and SSL enabled] *****************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************************************************************************************************************************************************************
ok: [ara-api-server]

TASK [Set ara.example.org to 127.0.0.1] ***************************************************************************************************************************************************************************************************************************************************************************************
changed: [ara-api-server]

TASK [Ensure directory for SSL certificate and private key exists] ************************************************************************************************************************************************************************************************************************************************************
changed: [ara-api-server] => (item=/etc/ssl/ara.example.org)
ok: [ara-api-server] => (item=/etc/ssl/ara.example.org)

TASK [Set path to selfsigned openssl configuration] ***************************************************************************************************************************************************************************************************************************************************************************
ok: [ara-api-server]

TASK [Template an openssl configuration file] *********************************************************************************************************************************************************************************************************************************************************************************
changed: [ara-api-server]

TASK [Generate private key and self-signed certificate] ***********************************************************************************************************************************************************************************************************************************************************************
changed: [ara-api-server]

TASK [Set up the API with the ara_api Ansible role] ***************************************************************************************************************************************************************************************************************************************************************************
ERROR! the role 'ara_api' was not found in /home/dmsimard/dev/git/ansible-community/ara-collection/tests/roles:/home/dmsimard/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/home/dmsimard/dev/git/ansible-community/ara-collection/tests

The error appears to be in '/home/dmsimard/dev/git/ansible-community/ara-collection/tests/with_nginx.yaml': line 90, column 15, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

      include_role:
        name: ara_api
              ^ here

PLAY RECAP ********************************************************************************************************************************************************************************************************************************************************************************************************************
ara-api-server             : ok=6    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

So the playbook ends without a failure (failed=0) but I think we can agree that the playbook actually failed.

Meanwhile, ara marks the include_role task as succeeded: https://demo.recordsansible.org/results/690240.html

Need to spend more time investigating to find out whether it's a bug in ara (are we missing a callback hook?) or if it's a bug in ansible-core (is it really firing a v2_runner_on_ok on a failed include_role task?).

What should be happening ?

The playbook should have been marked as failed because it did.

[DataHearth]

Hey there, I can investigate that tonight if you want ^^. I think (I don't know all ansible's hooks) this an O.K result (in terms of logic) because the playbook didn't failed any tasks here. So if there's a hooks for errors like this, it should be implemented of course. If not, maybe we can found a work around to check if there's such error during ansible execution.

[dmsimard]

I was curious about it and dug a little bit on the hypothesis I had.

It turns out that when an include_role task fails, it sends a v2_runner_on_ok callback hook instead of what should be v2_runner_on_failed because, well, it failed. This is probably part of why we don't see a failed=1.

I have a generic reproducer (with some ara debug logging turned on) in this gist: https://gist.github.com/dmsimard/8c77ea6db405fbf7b11e8f78f8818afb

I must say that ansible-playbook still exits non-zero (2, in fact) so it is properly a failure from the perspective of the shell.

I'll try to find out is there is already an issue about it in the ansible repo or I'll create a new one so we can consider fixing it upstream.