SSH known_hosts read_passphrase: can\'t open /dev/tty: No such device or address\r\nHost key verification failed

[emollusion]

Hi!

Not sure if this is a bug or if my config is wrong, but I have tried to both google it and different configurations so here goes:

SHA256****\r\ndebug3: hostkeys_foreach: reading file "/home/semaphore/.ssh/known_hosts"\r\ndebug1: read_passphrase: can\'t open /dev/tty: No such device or address\r\nHost key verification failed.\r\n')

The user on the server which is running semaphore, is named semaphore, hence the directory.

As I understand the error, the remote host is not in the known_hosts directory, and the ssh-agent is expecting input wether it should approve this remote host or not.

I have tried to mitigate this by telling ansible, and ssh, to not care about the host key by configuring the Environment.

{
  "inventory_hosts": "staging",
  "host_key_checking": "False",
  "ENV": {
    "ansible_ssh_common_args": "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
  }
}

In debug run it captures host_key_checking, but it does not seem to catch the ENV key.

I have not found anything relevant in the log on the server for semaphore.

Running this localy using ansible-playbook, as user semaphore, presented the same problem but when asking SSH to not care about host key check it just accepted the host keys and put these into /home/semaphore/.ssh/authorized_keys

So yes, a solution would be to keep that file updated, but that would require some more automation implemented for whenever the inventory is updated, feels like a work around but maybe it's the solution?

Any help appreciated!

Thanks!

[fiftin]

Hi @emollusion,

What Semaphore installation do you use (snap or binary) ?

[emollusion]

Hi!

I am using Semaphore 2.7.22 and run it as a binary.

[1024jon]

Hi, I am experiencing this issue as well. I was running docker version 2.7.22. However I am trying older versions now

Update: I didn't experience the issue on the snap version which is 2.7.22. I have the issue when running in docker containers. I've tried 2.8.5, 2.7.22, 2.6.9 and they all have the same issue.

Running with emollusion's environment file does not fix the issue for me.

[luismsousa]

@emollusion @1024jon Just came across the same issue. Were you able to overcome it?

[sagebrushIT]

No, I'm running the snap version for now which doesn't have the issue.

[fiftin]

Hi @emollusion

I can't reproduce the issue.

Did you try to add host_key_checking = False to your ansible.cfg ?

or

cat > /etc/ssh/ssh_config <<EOF
Host *
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null
EOF

[fiftin]

@luismsousa do you use binary or docker?

[luismsousa]

@luismsousa do you use binary or docker?

I was trying to use the docker images. @fiftin

[fiftin]

Unfortunately I can't reproduce the issue, I need help from Docker specialist.

[shizzgar]

@luismsousa, try to add this to your inventory file It can be set for whole inventory, not only for one host as in this example

[test]
my.ip.adr.ess ansible_ssh_extra_args='-o StrictHostKeyChecking=no'

[luismsousa]

@luismsousa, try to add this to your inventory file It can be set for whole inventory, not only for one host as in this example

[test]
my.ip.adr.ess ansible_ssh_extra_args='-o StrictHostKeyChecking=no'

yep, that did it @shizzgar I suspected it was hanging on accepting the key for the servers since the container doesn't have knowledge of your servers known_hosts file. Many thanks for confirming the theory!

[KasperSkytte]

Had this issue too. Was due to my ansible.cfg was placed in a subfolder of the git repo alongside the playbook. So it didn't read the config at all and setting host_key_checking = False there didn't have any effect. Moved everything to repo root and it all worked fine.

So I guess this is down to non-interactive sessions still thinking that they are interactive and prompting for a user input? Wouldn't it be possible to auto answer yes/continue etc on stuff like this with a setting? Similar to using DEBIAN_FRONTEND=noninteractive apt-get update -y when building containers non-interactively etc... just an example

[sweisgerber-dev]

@fiftin I've got this bug as well, when running semaphore in docker. host_key_checking = False solves the issue and connects and stores the host key in /home/semaphore/.ssh/known_hosts, also when adding the flags in the inventory file: ansible_ssh_common_args: '-o StrictHostKeyChecking=no'

There are multiple solutions and workarounds and I would advocate against the general use of host_key_checking = False.

@KasperSkytte & @fiftin I couldn't find any ssh flag, that accepts hostkeys of unknown hosts, but fails otherwise. Using the helper yes | ssh (....) would probably work, though.

My favorite solution, which isn't a workaround as it enables you a fine-grained control over the host keys, is to mount a known_hosts file from the host into the docker contain in read only mode, in my case the known_hosts from my server user.

# (...)
    volumes:
      - /srv/semaphore/playbooks/:/playbooks/
      - /root/.ssh/known_hosts:/home/semaphore/.ssh/known_hosts:ro
# (...)

But you can also hand-craft a separate known_hosts file with ssh-keyscan

Additional workaround:

Create an special inventory file just to learn host keys. Accept in the inventory all host Keys and use Ping as no op command: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/ping_module.html to connect to all remove hosts. In your actual playbooks, you then can leave host key checking active, as your known_hosts file is now filled.

[KasperSkytte]

@sweisgerber-dev thanks. Just went through possibly everything in the ssh man pages and seems there is just no way to "turn off" the interactiveness, at least for the host key checking part. I agree that disabling host key checking is not ideal, it's a security risk. The only way would be to manually add the host keys to ~/.ssh/known_hosts beforehand or as you suggest mount it from the host machine if running through docker. That's just the nature of SSH, security can be painful, but for good reason.

Would be handy if it was possible to simply add host keys to the ansible inventory or something similar. But that, on the other hand, would maybe be possible to do from semaphore in the same way as adding private ssh keys?

[1024jon]

FWIW, I'm running the newest version in docker(2.8.53) and I haven't run into this issue anymore...I just have "ANSIBLE_HOST_KEY_CHECKING: "false"" included in my docker-compose.

[sweisgerber-dev]

@1024jon Keep in mind that this disables a core security feature of SSH. It's the same as if your browser wouldn't check any authenticity of a website's TLS certificate and accepts all certificates (self-signed, wrong domains, etc).

[brileyd]

I had the same issue and not in docker. I found that if I added the "StrictHostKeyChecking no" entry to the ssh_config file and ran the play, I could then remove the same entry from the ssh_config file and the play still works. It is like it is running off a different known_hosts file than the OS user I created to run this under. After further research, I found that it is actually running under root (and that is the known_host file that gets updated when the entry is in the ssh_config file). I didn't do the semaphore installation myself and should have checked this sooner. Talking it out here actually helped me identify the issue...me. :) Thanks!

[cppmm]

I can confirm this issue with semaphore v2.8.79 in docker. If I enter into container using docker exec -it semaphore-semaphore-1 sh and create by hand /home/semaphore/.ssh/know_hosts file issue still exists. Only after connect to remote hosts with ssh -l username hostname from container and accepting host key by hand it works. But I have many hosts and adding every is problem. Workaround with disabling key checking is not very good I think, so it will be good to fix this issue. If any information or help from me is needed I ready to questions.