Cannot disable certificate validation when using vault_kv2_get lookup

[argetlam-coder]

SUMMARY

I am trying to retrieve a secret from my vault using the vault_kv2_get lookup. Since my vault uses a self-signed certificate, I need to disable certificate validation. I have set the corresponding parameter as described in the documentation but still the certificate validation is not deactivated.

ISSUE TYPE

• Bug Report

COMPONENT NAME

vault_kv2_get lookup

ANSIBLE VERSION

ansible [core 2.17.2]
  python version = 3.11.2 (main, Aug 26 2024, 07:20:54) [GCC 12.2.0]
  jinja version = 3.1.4

COLLECTION VERSION

Collection            Version
--------------------- -------
community.hashi_vault 6.2.0 

CONFIGURATION

[hashi_vault_collection]
auth_method = token
url = https://192.168.1.253:8200
retries = 5
token_path = ./
token_file = .vault-token

STEPS TO REPRODUCE

I have tried all three variants described in the documentation (https://docs.ansible.com/ansible/latest/collections/community/hashi_vault/vault_kv2_get_lookup.html#ansible-collections-community-hashi-vault-vault-kv2-get-lookup). Neither variables nor parameters nor environment variables worked.

Here is my playbook with variables

---
- name: vault test playbook
  hosts:
   - 127.0.0.1
  vars:
   ansible_hashi_vault_engine_mount_point: kv
   ansible_hashi_vault_validate_certs: false
  tasks:    
   - name: Perform a kv2 read with token login, showing the secret
     ansible.builtin.debug:
       msg: "{{ lookup('community.hashi_vault.vault_kv2_get', 'test')['secret'] }}"

and here is my playbook with parameters:

---
- name: vault test playbook
  hosts:
   - 127.0.0.1
  tasks:    
   - name: Perform akv2 read with token login, showing the secret
     ansible.builtin.debug:
       msg: "{{ lookup('community.hashi_vault.vault_kv2_get', 'test', engine_mount_point='kv', validate_certs=false)['secret'] }}"

EXPECTED RESULTS

The secret from the vault should be displayed.

ACTUAL RESULTS

I get the following error:

TASK [Perform akv2 read with a single Vault login, showing the secret] *********************************************************************************************************************
task path: /home/user/Repos/ansible-repository/openbao.yml:29
Loading collection community.hashi_vault from /home/user/Repos/ansible-repository/collections/ansible_collections/community/hashi_vault
[WARNING]: community.hashi_vault: 5 retries remaining.
[WARNING]: community.hashi_vault: 4 retries remaining.
[WARNING]: community.hashi_vault: 3 retries remaining.
[WARNING]: community.hashi_vault: 2 retries remaining.
[WARNING]: community.hashi_vault: 1 retry remaining.
exception during Jinja2 execution: Traceback (most recent call last):
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/connectionpool.py", line 466, in _make_request
    self._validate_conn(conn)
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1095, in _validate_conn
    conn.connect()
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/connection.py", line 730, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/connection.py", line 909, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 469, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 513, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/usr/lib/python3.11/ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:992)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/connectionpool.py", line 490, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:992)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/connectionpool.py", line 873, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/connectionpool.py", line 873, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/connectionpool.py", line 873, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.1.253', port=8200): Max retries exceeded with url: /v1/kv/data/test (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:992)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/ansible/template/__init__.py", line 856, in _lookup
    ran = instance.run(loop_terms, variables=self._available_variables, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Repos/ansible-repository/collections/ansible_collections/community/hashi_vault/plugins/lookup/vault_kv2_get.py", line 218, in run
    raw = client.secrets.kv.v2.read_secret_version(path=term, version=version, mount_point=engine_mount_point)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/hvac/api/secrets_engines/kv_v2.py", line 153, in read_secret_version
    return self._adapter.get(
           ^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/hvac/adapters.py", line 146, in get
    return self.request("get", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/hvac/adapters.py", line 408, in request
    response = super().request(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/hvac/adapters.py", line 367, in request
    response = self.session.request(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/ansible-core/lib/python3.11/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='192.168.1.253', port=8200): Max retries exceeded with url: /v1/kv/data/test (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:992)')))
fatal: [127.0.0.1]: FAILED! => {
    "msg": "An unhandled exception occurred while running the lookup plugin 'community.hashi_vault.vault_kv2_get'. Error was a <class 'requests.exceptions.SSLError'>, original message: HTTPSConnectionPool(host='192.168.1.253', port=8200): Max retries exceeded with url: /v1/kv/data/test (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:992)'))). HTTPSConnectionPool(host='192.168.1.253', port=8200): Max retries exceeded with url: /v1/kv/data/test (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:992)')))"
}

[briantist]

Hi @argetlam-coder , I didn't have a lot of time to dig too deeply into this yet so I'm not sure of the reason. We do have tests for that functionality that seem to be passing, see:

• https://github.com/ansible-collections/community.hashi_vault/actions/runs/11879488067/job/33101485417#step:12:1212
• https://github.com/ansible-collections/community.hashi_vault/actions/runs/11879488067/job/33101485417#step:12:1259
• https://github.com/ansible-collections/community.hashi_vault/blob/6.2.0/tests/unit/plugins/module_utils/test_hashi_vault_connection_options.py#L98
• https://github.com/ansible-collections/community.hashi_vault/blob/6.2.0/tests/integration/targets/connection_options/tasks/controller.yml#L60

I'm curious if any earlier versions of the collection work for you? I don't think we've changed that code in years though.

[argetlam-coder]

Hi @briantist , I will try an older version. In the meantime, I have tried the vault_kv2_get module. The module works with the following playbook:

---
- name: vault test playbook
  hosts:
   - 127.0.0.1
  vars:
   ansible_hashi_vault_engine_mount_point: kv
   ansible_hashi_vault_validate_certs: false
  tasks:
   - name: Read the latest version of a kv2 secret from Vault via the remote host with userpass auth
     community.hashi_vault.vault_kv2_get:
       url: https://192.168.1.253:8200
       token: xxxxxxxx
       engine_mount_point: kv
       path: test
       validate_certs: false
     register: response
   
   - name: Display the results
     ansible.builtin.debug:
       msg: "{{ response.secret }}"

It seems that only the pluginvault_kv2_get lookup plugin is affected by this problem.

[argetlam-coder]

I have now tried different versions (2.5.0, 3.4.0, 4.2.1, 5.0.1), but the problem persists in all versions. It works in all versions when I use the module, but not when I use the lookup plugin.

[argetlam-coder]

The error actually also occurs with the module. I had just accidentally configured the lookup plugin and the module differently.

The cause is the retries parameter. As soon as I set this, the certificate validation is executed regardless of how the validate_certs parameter is set.

I have found the error in the code and will create a pull request.

[maica1]

Same here, disabling the retries for now. It looks like with retries it ignores every way of setting off the certificate validation. It was really working before I set the retries at the ansible.cfg