Problem: SSH Connection refused after update to 2.13.10

[dzaegr]

Issue

Hi, after updating to latest (2.13.10, from 2.10.22) using Docker (with compose) I am receiving SSH Connection Refused errors when running Ansible Task Templates. Before the update all the Playbooks ran well. I checked the Key Store, and the inventory. All configured correctly, I manually checked the SSH Key, it still works. I tried setting up a new project, replicating the settings, still not working.

I'm running out of ideas.

Any configuration updates I might have missed? In the documentation, I could not find any hints, except the removal of SSH Option Overwriting, which I didn't use before, so this should not be the issue.

After I got the error, I also checked the docker hub page, where I saw the latest version was updated. I tried updating to that as well, still getting the issue.

I also checked the SSH logs on the target server, which actually showed successful connection attempts, but it also stated that the connection was closed by the client.

Another thing to add: It also only seems to impact tasks which would lead to "changed" status. "skipped" and "ok" tasks in the playboook actually don't produce the error.

Any ideas?

Impact

Ansible (task execution)

Installation method

Docker

Database

MySQL

Browser

No response

Semaphore Version

2.13.10 and later 2.13.12

Ansible Version

2.18.4 (in docker container)

Logs & errors

10:38:26 PM msg: 'Failed to connect to the host via ssh: ssh: connect to host HOSTNAME port 22: Connection refused'

Manual installation - system information

No response

Configuration

No response

Additional information

No response

[dzaegr]

I was able to narrow it down at least a little bit. After downgrading to 2.13.0 it worked for me again (first version with last migration version).

[fiftin]

Hi @dzaegr do you see any errors in Semaphore log?:

docker logs -f semaphore_container_name

[fiftin]

Hi @dzaegr Only important difference which I found by code between 2.13.0 and 2.13.1 is line in ansible.cfg in docker container.

In 2.13.0:

[defaults]
host_key_checking = False
bin_ansible_callbacks = True
stdout_callback = yaml

In 2.13.1:

[defaults]
bin_ansible_callbacks = True
stdout_callback = yaml

Can you manually add this line (host_key_checking = False) to /etc/ansible/ansible.cfg in your Semaphore 2.13.10 in container?

[dzaegr]

Hi @dzaegr do you see any errors in Semaphore log?:

docker logs -f semaphore_container_name

Only some errors regarding abnormal closure of the websocket, but this already occured before the update. (And actually still regularly occurs today).

Can you manually add this line (host_key_checking = False) to /etc/ansible/ansible.cfg in your Semaphore 2.13.10 in container?

Does not work unfortunately. But I provide this option as an environment variable anyway, so I guess this should not make a difference to Ansible.

I also tried modifying the ansible.cfg file and omitting the environment variable, but this doesn't work either.

[dzaegr]

I just tried version 2.13.1, with and without host_key_checking = False. So the problem came with this version.

[fiftin]

This line was added in 2.13.1:

But it was removed in 2.13.5.

[fiftin]

Hi @dzaegr

The issue is actual?

[dzaegr]

Hi @fiftin, I just tried with the latest docker version (v2.13.14-510ae35). On my local machine it seems to work, though I can't really tell what actually fixed the issue. I did not modify any files in the container after the compose build, so it actually works with the image out of the box.

I will try on our live server, when I find the time, but for now I think the issue is resolved.

Thanks for your help! :)

[gschafra]

I'd also experienced a SSH connection problem, which seems to be introduced with any of the versions between v2.13.0 and v2.13.14:

4:39:51 PM
TASK [Wait until SSH port 22 is open] ******************************************
4:40:01 PM
fatal: [ansible-docker]: UNREACHABLE! => changed=false 
4:40:01 PM
  msg: 'Failed to connect to the host via ssh: ssh: connect to host 88.99.125.24 port 22: Operation timed out'
4:40:01 PM
  unreachable: true

After dropping lines in the [ssh_connection]-Block from /etc/ansible/ansible.cfg with in the container, everything seems to work:

sed -i '/^\[ssh_connection\]/,/^$/d' /etc/ansible/ansible.cfg

I'll try to investigate further tomorrow...

[GorgetGuilhem]

Same issue after upgrading to v2.13.14 yesterday. Oddly, this problem seems to affect python 3.12 only. My 22.04 ubuntu machines working with python 3.10 are still reachable while all ubuntu 24.05 machines which includes python 3.12 arn't.

Dropping the [ssh_connection] block from /etc/ansible/ansible.cfg in the semaphore container works for me though.

[fiftin]

Hi @GorgetGuilhem thank you.

It is real problem, because this block has been introduced to fix other issue with SSH access.

[fiftin]

@GorgetGuilhem is it actual for 2.16?