semaphore icon indicating copy to clipboard operation
semaphore copied to clipboard

target unreachable

Open chaddyc opened this issue 2 years ago • 30 comments

I set up everything all my ssh keys and tested everything manually and have been running ansible for a while now. However, when I do the same with Ansible Semaphore I get a "target unreachable" error but all ssh keys etc are set up? Any advice?

chaddyc avatar Jul 10 '22 20:07 chaddyc

Same problem

mac-lucky avatar Sep 02 '22 23:09 mac-lucky

Hello, same problem for me, Is it solved for you?

EpicShiield avatar Nov 02 '22 08:11 EpicShiield

Hi @chaddyc , @mac-lucky , @EpicShiield if you solved the issue, how you do this?

fiftin avatar Jul 09 '23 10:07 fiftin

Same, I was unable to find a way to get debug logs, this might help. But my 3 hosts all failed, 2x local and 1x public IP

  • all Ubuntu 22.04 hosts
  • despite being able to each out and sync with GitHub....

Screenshot 2023-07-31 at 5 55 51 pm

adamphetamine avatar Jul 31 '23 08:07 adamphetamine

Hi @fiftin, i’ve solved it but i didn’t remember how, i’m sorry, i’ll try to reproduce it when i can and if i can

EpicShiield avatar Jul 31 '23 08:07 EpicShiield

@adamphetamine, is it a new Install ?

EpicShiield avatar Jul 31 '23 08:07 EpicShiield

Yes, this is a brand new install. Might be able to fix it myself if I could get debug logs...

adamphetamine avatar Jul 31 '23 23:07 adamphetamine

This has been an issue. for me as well. I've found two ways to resolve this.

The first is to create a config file in the container here /home/semaphore/.ssh/config and put the following lines to it to have it skip checking the host keys (This is not a safe practice though).

Host *
   StrictHostKeyChecking no

The next is to add each host key to /home/semaphore/.ssh/known_hosts this is also a problem because any time a new host is added to the inventory the host's respective key will also need to be added.

Neither of these persists when the container is recreated or updated and has to be redone each time the container is recreated. At the moment this makes semaphores in a container almost useless.

jgranzow86 avatar Aug 01 '23 16:08 jgranzow86

Thanks @jgranzow86 for the fix and the explanation, this is great.

It also infers that Semaphore could be 'fixed' if it managed the 'known hosts' as part of Inventory setup- is this correct?

adamphetamine avatar Aug 02 '23 00:08 adamphetamine

ok I figured it out. Undoubtedly there is some security and usability concerns but this works-

In your Semaphore project folder create a directory called ssh

add a text file inside that folder called known_hosts and add the details from these hosts

Now add this to your docker-compose.yaml in the semaphore section-

volumes:
  - ./ssh/known_hosts:/home/semaphore/.ssh/known_hosts

Now you can start the project and it will connect to your SSH nodes without issues Thanks to @jgranzow86 for the tip!

adamphetamine avatar Aug 02 '23 07:08 adamphetamine

I'm really happy that will work for you. Unfortunately for my setup I've got an environment with well over 1000 VM's and devices. Maintaining that file by hand will be almost impossible for me. We need a proper solution put in place by the developers.

jgranzow86 avatar Aug 02 '23 11:08 jgranzow86

Agreed, but I added the workaround in part to encourage the dev to look at a fix. I'm not smart enough to do it myself, but I'm hoping this is easy and can be done quickly

adamphetamine avatar Aug 03 '23 00:08 adamphetamine

Hello, i've added this in docker compose file and that works after: ANSIBLE_HOST_KEY_CHECKING: "False"

Kélian

EpicShiield avatar Aug 08 '23 11:08 EpicShiield

Hello, i've added this in docker compose file and that works after: ANSIBLE_HOST_KEY_CHECKING: "False"

Kélian

While that does resolve the issue. it creates a huge security issue since key's won't be checked against the known list. That's a massive problem

jgranzow86 avatar Aug 08 '23 23:08 jgranzow86

Hello, i've added this in docker compose file and that works after: ANSIBLE_HOST_KEY_CHECKING: "False" Kélian

While that does resolve the issue. it creates a huge security issue since key's won't be checked against the known list. That's a massive problem

Hello,

You do the same thing with your solution, no ?

Host *
   StrictHostKeyChecking no

Kélian

EpicShiield avatar Aug 09 '23 08:08 EpicShiield

You do the same thing with your solution, no ?

I did only as a test to show that it's an issue with validating the keys. I turned it off since verifying it's the issue. I also made note of it being dangers to bypass host key checking. This is one of those things that you do at your own risk. The key checking is there for a reason and the creators of SSH thought it was important enough to enable by default to increase security. So in my opinion I only disable it where I truly have to. With something as important as Ansible configuring my systems I want it to be quite secure.

jgranzow86 avatar Aug 09 '23 08:08 jgranzow86

ok I see, thanks for the explanation, so I'll proceed as you did with a know_hosts file, but isn't it laborious to maintain?

EpicShiield avatar Aug 09 '23 08:08 EpicShiield

so I'll proceed as you did with a know_hosts file, but isn't it laborious to maintain?

It's quite laborious, this issue is something the devs need to fix. It's so tedious that I'm not even going to use the software until some kind of resolution is made. I'm looking at other software right now. I wanted Semaphore to work so badly but with this issue it's a show stopper for me.

jgranzow86 avatar Aug 09 '23 09:08 jgranzow86

I see, I think he wants to solve this problem but needed more information on the issue, now I hope he does 😄

EpicShiield avatar Aug 09 '23 09:08 EpicShiield

hi @jgranzow86 & @adamphetamine, i tried to bind a known hosts file with a host fingerprint but when i ran a task in semaphore, i've permission denied to access to this file image

do you have any idea ?

Kélian

EpicShiield avatar Aug 16 '23 08:08 EpicShiield

When i'm going in the semaphore container with docker exec, i can see the .ssh folder, but he don't have the good rights, image

how can i modify this ? Kélian

EpicShiield avatar Aug 16 '23 13:08 EpicShiield

I have the same issue with permission denied

tbbkitty avatar Jan 19 '24 01:01 tbbkitty

I have the same issue with permission denied

did you tried the solution of @adamphetamine ?

ksourdrille avatar Jan 19 '24 09:01 ksourdrille

I have 50 hosts and I will not accept such a manual process for a service intended to automate tasks as a solution. It is at best a temporary workaround. But yes it works for one host

tbbkitty avatar Jan 19 '24 13:01 tbbkitty

I have 50 hosts and I will not accept such a manual process for a service intended to automate tasks as a solution. It is at best a temporary workaround. But yes it works for one host

I understand, but there's always the unsecured solution.... i've no best solution atm, and i don't know if someone have one

ANSIBLE_HOST_KEY_CHECKING: "False"

ksourdrille avatar Jan 19 '24 13:01 ksourdrille

Update on that, if you disable hostkey checking, it will ad the host to your well-known and then you can re enable it again

tbbkitty avatar Jan 19 '24 13:01 tbbkitty

Update on that, if you disable hostkey checking, it will ad the host to your well-known and then you can re enable it again

are you sure about that ? i'll try

ksourdrille avatar Jan 19 '24 13:01 ksourdrille

Update on that, if you disable hostkey checking, it will ad the host to your well-known and then you can re enable it again

Not for me,

i've disabled the host key checking and recreated the container ran a playbook on 2 hosts re enable the host key checking and recreated the container ran the same playbook and "Failed to connect to the host via ssh: Host key verification failed."

Did you do something else ?

ksourdrille avatar Jan 19 '24 14:01 ksourdrille

Update on that, if you disable hostkey checking, it will ad the host to your well-known and then you can re enable it again

Not for me,

i've disabled the host key checking and recreated the container ran a playbook on 2 hosts re enable the host key checking and recreated the container ran the same playbook and "Failed to connect to the host via ssh: Host key verification failed."

Did you do something else ?

Seems like it might just have been a fluke, I have for now just decided to leave key verification disabled, which I know is a terrible idea, but I am giving up after 4 hours of troubleshooting. I hope they look into this soon.

tbbkitty avatar Jan 21 '24 23:01 tbbkitty

Update on that, if you disable hostkey checking, it will ad the host to your well-known and then you can re enable it again

Not for me, i've disabled the host key checking and recreated the container ran a playbook on 2 hosts re enable the host key checking and recreated the container ran the same playbook and "Failed to connect to the host via ssh: Host key verification failed." Did you do something else ?

Seems like it might just have been a fluke, I have for now just decided to leave key verification disabled, which I know is a terrible idea, but I am giving up after 4 hours of troubleshooting. I hope they look into this soon.

Hello, thank you for your feedback, yes for me it's disabled too but it's not a good idea... because I can't manually keep a file up to date :/.

I hope the developer will look into this.

ksourdrille avatar Jan 22 '24 08:01 ksourdrille