AWX does not check SSH host keys

[azrdev]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

Since AWX runs jobs in throw-away containers, the ~/.ssh/known_hosts file has to be persisted outside the container to be of use. Currently, the ssh option -o StrictHostKeyChecking doesn't matter because old hostkeys are not known, so AWX happily accepts any host keys.

AWX version

21.9.0

Select the relevant components

• [ ] UI
• [ ] API
• [ ] Docs
• [ ] Collection
• [ ] CLI
• [X] Other

Installation method

kubernetes

Modifications

no

Ansible version

2.12.5.post0

Operating system

No response

Web browser

No response

Steps to reproduce

1. Add a host in AWX that is accessed over SSH
2. run a job on that host
3. login to that host otherwise and force recreation of host keys. e.g. on centos 8: sudo rm /etc/ssh/ssh_host_* sudo systemctl start sshd-keygen@ed25519 sudo systemctl restart sshd
4. run another AWX job on that host

Expected results

the new job to fail because of changed hostkeys

Actual results

job is ok

Additional information

#2282 and #452 read similar, but were before EEs

[djyasin]

Hi @ffirg! Do you have any insight for a workaround for this instance?

[ffirg]

Perhaps have a centralised ~/ssh/known_hosts and use Job isolation paths to ensure this is mounted on every execution node?

From /etc/ssh/ssh_config docs: UserKnownHostsFile Specifies a file to use for per-user known host key database instead of the default ~/.ssh/known_hosts

[azrdev]

IMHO the proper way to solve it would be to have the known hostkeys as a property of each host in the AWX database. After fact gathering/caching awx could then extend this field with all found (e.g. changed hostkeys, or new algorithms).

This would probably require

• a property of the host
• CRUD API + GUI for that property
• a setting whether to trust unkown hostkeys (aka TOFU / ssh -oStrictHostKeyChecking=allow-new)
• code to add discovered hostkeys to that property when a playbook runs with that kind of fact gathering
• (maybe) a setting whether to add discovered hostkeys to the property

[Mr-Philipp]

This should be addressed asap because as it is implemented right now security is basically non-existent.

azrdev already has some good ideas on what might be necessary. until then a workaround could be building a custom Execution Environment Image with backed in trusted hosts keys.

[Ronaldr1985]

Hey guys,

Is there any news on this? Also, currently does AWX explicitly set strict_host_key_checking=no anywhere?