Sync Job Fails on Galaxy Collection Install: Cert Validate Failed

[Dustin-Wi]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
• [X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to [email protected] instead.)

Bug Summary

When syncing a job, if I include a collection/requirements.yml file the sync fails with the below error.

I suspect this happens because this is installed in our enterprise environment that uses a self-signed cert on all internet destined HTTPS communications at our web gateways/proxies.

[WARNING]: Skipping Galaxy server https://galaxy.ansible.com/. Got an unexpected error when getting available versions of collection community.crypto: Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)> ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>

AWX version

23.4.0

Select the relevant components

• [ ] UI
• [ ] UI (tech preview)
• [ ] API
• [ ] Docs
• [ ] Collection
• [ ] CLI
• [X] Other

Installation method

kubernetes

Modifications

no

Ansible version

2.15.6

Operating system

Red Hat Enterprise Linux release 8.8 (Ootpa)

Web browser

Firefox

Steps to reproduce

Install AWX-Operator 2.7.2 using rancher k3s Sync a project from source control that has a requirements.yml file containing collections to install.

Expected results

The project should sync. AWX should install the collection.

Actual results

Job sync fails, complaining about certificate validation with the below error on the Project Sync job details in AWX. [WARNING]: Skipping Galaxy server https://galaxy.ansible.com/. Got an unexpected error when getting available versions of collection community.crypto: Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)> ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>

Additional information

I am able to work around this for non-standard collections like a10.acos_axapi by installing them to the collections folder in the project and using ansible.cfg to specify the collection directory. However, I cannot get some of the collections that come with CLI ansible to install to that collection folder.

[Dustin-Wi]

I researched for over and hour to find a fix or workaround for all collections, and it wasn't till I submitted this issue that I found a workaround.

For collections that are installed with "pip install ansible" but not in AWX, if I just re-install the collection it installs to the specific collection folder in the project.

ansible-galaxy collection install -U <collection_name>

I'm still interested in a solution to allow the Project to Sync using collection requirements.yml

[djyasin]

@Dustin-Wi We are so happy to hear you were able to find a workaround!

[cvvs]

We're impacted by this, and the provided workaround isn't really an option for us.

[DustinWi]

See this thread in the Ansible forum. https://forum.ansible.com/t/error-in-kustomization-yml-unknown-field-spec-containers/4453

My permanent solution was to do as Kurokobo and Denney-Tech suggested, follow the instructions in this link: https://ansible.readthedocs.io/projects/awx-operator/en/latest/user-guide/advanced-configuration/trusting-a-custom-certificate-authority.html

1. Create a kubernetes custom-ca-bundl secret
2. Reference the newly created secret in your awx operator kustomization file.

# Sample of creating the secret in your kustomization.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
secretGenerator:
  - name: my-custom-certs
    files:
      - bundle-ca.crt=custom-ca-bundle.crt
    options:
      disableNameSuffixHash: true

resources:
  # Add your Resources

# Set the image tags to match the git version from above
images:
  # Add your AWX Images

# Specify a custom namespace in which to install AWX if you want to
namespace: awx

# Sample of referencing the newly created secret in your awx kustomization file
# Filename: awx.yaml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # Include your other spec info as well
  bundle_cacert_secret: my-custom-certs

[cvvs]

We had already done that, but I added an additional cert to the bundle and retried. While doing a triple-check confirm that it wasn't working, it miracled itself fixed, I guess. I assumed I could just update the secret and do a rollout restart of the deployment, but that didn't seem to work... until it did.

That's definitely the fix, but it may take a while.

[DustinWi]

In my case, I had a hard time getting it to work for two reasons:

1. I accidentally created the k3s secret on the CLI and in the awx.yaml file, causing conflicts.
2. I renewed my certificates, but used the same file name.

Mine didn't work until after I deleted the secrets and used a new file name. I'm assuming changing the filename forced K3S to rebuild the secret, instead of it thinking the current secret is accurate and up to date. I'm not able to recreate the issue, so I don't really know with 100% certainty though.