Ansible vault not decrypted in httpapi after include_vars or set_fact

[diLLec]

SUMMARY

While using httpapi with the collection fortinet.fortios (version 1.0.11) the modules will report quote_from_bytes() expected bytes if used after include_vars or set_fact, because the "password" option of httapi will contain the vault dict instead of the actual password when self.httpapi.login(self.get_option('remote_user'), self.get_option('password')) is called in lib\ansible\plugins\connection\httpapi.py:241

ISSUE TYPE

• Bug Report

COMPONENT NAME

connection plugin -> httpapi

ANSIBLE VERSION

ansible 2.9.9.post0
  config file = None
  configured module search path = ['/home/mky/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/mky/ansible_devel/ansible/lib/ansible
  executable location = /home/mky/ansible_devel/ansible/bin/ansible
  python version = 3.6.8 (default, Jan 14 2019, 11:02:34) [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]

CONFIGURATION

$ ansible-config dump --only-changed
<none>

OS / ENVIRONMENT

Ubuntu in WSL

$ python --version
Python 3.6.8

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic

STEPS TO REPRODUCE

inventory

targets:
    hosts:
        testfw:
            ansible_host: "172.30.30.30"
            persistent_log_messages: "yes"
            ansible_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          36633363346231366462633433353430363030313033333234323961393935633636343933626663
          6333386536653362346632383364306431393937353063300a333636623634636336356639643465
          64376662396261656632396236636636656338643464363539646366326537343062303462613032
          3833353862393561300a373936376163386435316432646436646631323733633562356133346635
          3963

playbook

---
- name: "test"
  hosts: all
  gather_facts: False
  collections:
    - fortinet.fortios
  become: false
  connection: httpapi
  tasks:
    - set_fact:
        some_random_fact: true
    - name: "ensure virtual domain"
      fortios_system_vdom:
        vdom:  "global"
        system_vdom:
          state: "present"
          name: "root"
          short_name: "root"

EXPECTED RESULTS

If set_fact is obtained - the result is:

TASK [ensure virtual domain] 
ok: [cong-itffm-afw01]

Note: The same behavior can be seen if assert or include_vars is used

    - name: "check variables on asset"
      tags: always
      assert:
        that:
          - ansible_user is defined
          - ansible_httpapi_password is defined or ansible_password is defined

ACTUAL RESULTS

$ ANSIBLE_CONFIG=ansible.cfg ansible-playbook test.yml --vault-password-file  ~/test  -i inventory/inventory.yml -l testfw

fatal: [testfw]: FAILED! => changed=false
  ansible_facts:
    discovered_interpreter_python: /usr/bin/python
  module_stderr: |-
    Traceback (most recent call last):
      File "/home/mky/.ansible/tmp/ansible-local-242561mudj7lh/ansible-tmp-1591695232.2185314-24274-201421478678594/AnsiballZ_fortios_router_static.py", line 102, in <module>
        _ansiballz_main()
      File "/home/mky/.ansible/tmp/ansible-local-242561mudj7lh/ansible-tmp-1591695232.2185314-24274-201421478678594/AnsiballZ_fortios_router_static.py", line 94, in _ansiballz_main
        invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
      File "/home/mky/.ansible/tmp/ansible-local-242561mudj7lh/ansible-tmp-1591695232.2185314-24274-201421478678594/AnsiballZ_fortios_router_static.py", line 40, in invoke_module
        runpy.run_module(mod_name='ansible_collections.fortinet.fortios.plugins.modules.fortios_router_static', init_globals=None, run_name='__main__', alter_sys=True)
      File "/usr/lib/python3.6/runpy.py", line 205, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/usr/lib/python3.6/runpy.py", line 96, in _run_module_code
        mod_name, mod_spec, pkg_name, script_name)
      File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/tmp/ansible_fortios_router_static_payload_4ojwxm5_/ansible_fortios_router_static_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_router_static.py", line 487, in <module>
      File "/tmp/ansible_fortios_router_static_payload_4ojwxm5_/ansible_fortios_router_static_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_router_static.py", line 455, in main
      File "/tmp/ansible_fortios_router_static_payload_4ojwxm5_/ansible_fortios_router_static_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_router_static.py", line 382, in fortios_router
      File "/tmp/ansible_fortios_router_static_payload_4ojwxm5_/ansible_fortios_router_static_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_router_static.py", line 365, in router_static
      File "/tmp/ansible_fortios_router_static_payload_4ojwxm5_/ansible_fortios_router_static_payload.zip/ansible_collections/fortinet/fortios/plugins/module_utils/fortios/fortios.py", line 173, in set
      File "/tmp/ansible_fortios_router_static_payload_4ojwxm5_/ansible_fortios_router_static_payload.zip/ansible_collections/fortinet/fortios/plugins/module_utils/fortios/fortios.py", line 146, in get_mkey
      File "/tmp/ansible_fortios_router_static_payload_4ojwxm5_/ansible_fortios_router_static_payload.zip/ansible_collections/fortinet/fortios/plugins/module_utils/fortios/fortios.py", line 137, in get_mkeyname
      File "/tmp/ansible_fortios_router_static_payload_4ojwxm5_/ansible_fortios_router_static_payload.zip/ansible_collections/fortinet/fortios/plugins/module_utils/fortios/fortios.py", line 126, in schema
      File "/tmp/ansible_fortios_router_static_payload_4ojwxm5_/ansible_fortios_router_static_payload.zip/ansible/module_utils/connection.py", line 185, in __rpc__
    ansible.module_utils.connection.ConnectionError: quote_from_bytes() expected bytes
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

[diLLec]

As noted in the summary, I've tracked the problem to the following line in lib\ansible\plugins\connection\httpapi.py:241

self.httpapi.login(self.get_option('remote_user'), self.get_option('password'))

The get_options originates in lib/ansible/plugins/__init__.py. Putting the following there, will show that in the "EXPECTED RESULTS" case the decrypted password is returned (...'password': 'test',...)and in the "ACTUAL RESULTS" the vault dict is returned ('password': {'__ansible_vault': '$ANSIBLE_VAULT;1....)

        import traceback
        if option == 'password':
            raise Exception("%s -- %s" % (self._options, traceback.format_stack()))

From the other connection plugins, I've already seen that self._play_context is used. I've replace the line with

            self.httpapi.login(self._play_context.remote_user, self._play_context.password)

which worked. As I don't know the side effects of this, please review :-).

[yarokifor]

I could be wrong, but the issue looks like it is in module_utils/connection.py on line 142:

data = json.dumps(req, cls=AnsibleJSONEncoder)

Should vault_to_text=true argument be passed in here?

[yarokifor]

Looks like there's already an issue with ansible.

https://github.com/ansible/ansible/issues/75503

[gmuloc]

indeed - should have been fixed by: https://github.com/ansible/ansible/pull/78236

[KB-perByte]

Closing it off, as the issue is already fixed in https://github.com/ansible/ansible/pull/78236 Regards