Issue with AWX Callback and Host Matching Error

[analissec]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
• [X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to [email protected] instead.)

Bug Summary

I'm encountering a problem with the AWX callback command and host matching. When I run the Ansible callback command: /usr/bin/curl -k --data "host_config_key=test-ansible-host" https://[AWX_SERVER]/api/v2/job_templates/1527/callback/

I receive the following response: {"msg":"No matching host could be found!"}

Here's some additional context:

• I've followed the documentation to integrate my Satellite hosts with AWX, and they are successfully picked up by the dynamic inventory.
• However, when running the callback command, only an inventory sync is triggered, and then I encounter the "no matching hosts" error, despite my host being present in the inventory.
• I've configured REMOTE_HOST_HEADERS = ['HTTP_X_FORWARDED_FOR', 'REMOTE_ADDR', 'REMOTE_HOST'] in my AWX configuration.
• Interestingly, when I manually run the playbook from AWX on the same host I'm trying to use the callback on, it works without any issues.

Any insights or assistance on resolving this issue would be greatly appreciated!

AWX version

22.3.0

Select the relevant components

• [ ] UI
• [ ] UI (tech preview)
• [ ] API
• [ ] Docs
• [ ] Collection
• [ ] CLI
• [ ] Other

Installation method

N/A

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

• Ensure that AWX is properly configured and running.
• Integrate Satellite hosts with AWX using the documentation.
• Verify that the Satellite hosts are successfully picked up by the dynamic inventory in AWX.
• Execute the Ansible callback command as mentioned above, substituting [AWX_SERVER] with your AWX server's URL and the job template callback URL.

Expected results

Upon running the Ansible callback command, the system should successfully identify and match the host in the AWX dynamic inventory. The command should then trigger the specified job template on the identified host without encountering any errors. Additionally, the callback should execute any associated tasks or actions as intended.

Actual results

Upon running the Ansible callback command, the system fails to identify and match the host specified in the AWX dynamic inventory. Instead, it returns an error message stating "No matching host could be found!" despite the host being present in the inventory.

Additional information

No response

[thedoubl3j]

hey @analissec are you able to reproduce this on the latest version of awx?

[tonythell]

I am running v24.4.0 and am having this same issue.

$ curl -k -H 'Content-Type: application/json' -d '{"host_config_key": "test-awx-callback"}' https://[AWX Server]/api/v2/job_templates/15/callback/
{"msg":"No matching host could be found!"}

Like @analissec, I, too, am importing hosts from Red Hat Satellite using the Satellite inventory plugin. However, I can successfully run a job template directly against the client without issue.

[chrismeyersfsu]

@analissec @tonythell are ya'll using the PROXY_IP_ALLOWED_LIST setting?

It can cause the headers listed in REMOTE_HOST_HEADERS to be deleted from the request.

[tonythell]

@chrismeyersfsu The PROXY_IP_ALLOWED_LIST setting is an empty list in my config.

[chrismeyersfsu]

https://[AWX Server]/api/v2/job_templates/15/callback/ <-- issue a GET request with settings.DEBUG = True on to see if the forwarded for header is making it through and that the value is as expected.

{
    "host_config_key": "hello_world",
    "matching_hosts": [],
    "request_meta": {
        "REMOTE_ADDR": "192.168.96.1",
        "REMOTE_PORT": "58598",
        "HTTP_HOST": "localhost:8043",
        "HTTP_CONNECTION": "keep-alive",
        "HTTP_SEC_CH_UA": "<redacted>",
        "HTTP_SEC_CH_UA_MOBILE": "?0",
        "HTTP_SEC_CH_UA_PLATFORM": "\"Linux\"",
        "HTTP_UPGRADE_INSECURE_REQUESTS": "1",
        "HTTP_USER_AGENT": "<redacted>",
        "HTTP_ACCEPT": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
        "HTTP_SEC_FETCH_SITE": "same-origin",
        "HTTP_SEC_FETCH_MODE": "navigate",
        "HTTP_SEC_FETCH_USER": "?1",
        "HTTP_SEC_FETCH_DEST": "document",
        "HTTP_REFERER": "https://localhost:8043/api/v2/job_templates/8/",
        "HTTP_ACCEPT_ENCODING": "gzip, deflate, br, zstd",
        "HTTP_ACCEPT_LANGUAGE": "en,en-US;q=0.9",
        "HTTP_COOKIE": "<redacted>"
    }
}

[tonythell]

Could you please clarify the settings.DEBUG = True setting. I'm not sure if that is meant to be passed in with the actual GET request or is an AWX setting.

[chrismeyersfsu]

AWX setting

[tonythell]

Thank you for your help @chrismeyersfsu.

I found that my issue wasn't an actual AWX issue. I'm running awx-operator in a single-node k3s cluster and found that I was missing an additional Traefik config to allow X-Forwarded-For headers.

Essentially, I followed the instructions found here (https://github.com/kurokobo/awx-on-k3s/blob/main/tips/troubleshooting.md#provisioning-callback-does-not-work).

After that change, provisioning callbacks started working for me.