Docker Outpost integration via SSH not working

[RXWatcher]

Describe your question/ Authentik 2022.9.0.

Am I properly configuring the remote docker outpost deployment using ssh?

I am attempting to use the automatic docker outpost deployment on a remote machine from this doc: https://goauthentik.io/docs/outposts/integrations/docker

I reviewed the code: https://github.com/goauthentik/authentik/blob/main/authentik/outposts/docker_ssh.py and https://github.com/goauthentik/authentik/blob/main/authentik/outposts/controllers/docker.py

I'm seeing it do a pre-auth on the remote server when I review the /var/log/auth.log but it never actually tries to log in with a user.

I did add the public key generated per the doc to my users authorized_keys and it has the proper permissions on the remote host.

docker url: ssh://[email protected] I also tried ssh://[email protected]:22

I have tried:

1. Generating the certs per the doc and importing them into Authentik's certificates and created the outpost and without mapping them internally. This didnt work and I didnt see the certs in /authentik/.ssh/config
2. mapping the certs to /authentik/.ssh/config. The actual generated cert files authenik, authentik.pub and certificate.pem are in the /authentik/.ssh/config directory with permissions of 700. I confirmed the files are in place by docker execing into the container.
3. since I was using my own user in the docker ssh url, I tried removing that, creating an account on the remote server with a user of 'authentik' and the proper authorized_keys and it still did not work.

I have turned on debug logging in authentik to try to capture the ssh transaction but I'm unable to see where its even trying. If I grep for my hostname, my_user, etc then nothing is found. Where would I look to further diagnose this?

Relevant infos Ubuntu 20.04 all around public key auth ssh enabled on all machines, password ssh removed.

Thanks for your time.

[BeryJu]

/authentik/.ssh/config should be an SSH config file, not a directory. This is because the SSH Client used by the python docker SDK uses paramiko, but doesn't allow you to pass anything, so authentik writes that config to point to temporary key files

(Also all of this only happens on the worker, not the server)

For the actual logging in, I don't know, I would assume it just uses SSH Port-forwarding so it won't actually open a session

[RXWatcher]

thanks for pointing me to the worker. I am seeing the /authentik/.ssh/config being created on the worker and pointing towards the private key in /dev/shm so that part appears to working.

now to see if I can capture the actual ssh session with what user it's using and why it's failing.

[RXWatcher]

I solved this. Its a bug.

I had to map the ssh and vi binaries into the worker container so I could actually try a interactive ssh session. I then edited /etc/passwrd to allow the authentik user an interactive shell and did a su - authentik within the worker container. I then tried to ssh into my remote server as authentik and I got this:

The authenticity of host 'host.domain.com (10.10.10.8)' can't be established.
ECDSA key fingerprint is SHA256:4arMhcMfymZflaDdE5om5cuRLVbwWqttmJONoSDO0ZI.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'host.domain.com (10.10.10.8)' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 20.04.5 LTS (GNU/Linux 5.19.11-xanmod1 x86_64)

Once I accepted that I was let right into the server without password so the generated keys works properly to let me in.

At this point I went back to Authentik and tried the docker outpost integration, it worked and when I tried to deploy an outpost on the remote host it worked.

So Authentik needs to somehow make it so the ssh doesnt get asked to accept the remote hosts fingerprint.

[BeryJu]

Huh, I wonder how that was never noticed previously

[RXWatcher]

https://docs.paramiko.org/en/stable/api/client.html https://docs.paramiko.org/en/stable/api/client.html#paramiko.client.MissingHostKeyPolicy

If the server’s hostname is not found in either set of host keys, the missing host key policy is used (see [set_missing_host_key_policy](https://docs.paramiko.org/en/stable/api/client.html#paramiko.client.SSHClient.set_missing_host_key_policy)). The default policy is to reject the key and raise an [SSHException](https://docs.paramiko.org/en/stable/api/ssh_exception.html#paramiko.ssh_exception.SSHException).

It looks like you need to add the set_missing_host_key_policy policy to autoadd.

[BeryJu]

sadly the docker SDK doesn't allow setting that, but this PR should fix this https://github.com/goauthentik/authentik/pull/3691

[RXWatcher]

I'm testing this and it's now not creating the pem file in /dev/shm but /authentik/.ssh/config says it's there

### Managed by authentik - server.domain.com
Host server.domain.com
    IdentityFile /dev/shm/3d522c9e-749e-4c20-991a-df4f27963ac8_private.pem
   StrictHostKeyChecking No
   UserKnownHostsFile /dev/null
### End Managed by authentik

root@7bd0ff10e118:/authentik/.ssh# ls -al /dev/shm
total 0
drwxrwxrwt 3 root      root       60 Sep 28 12:49 .
drwxr-xr-x 5 root      root      340 Sep 28 12:42 ..
drwx------ 2 authentik authentik  40 Sep 28 12:42 pymp-2k_i8ooq

you're getting these errors in the worker log:

5d1fae0e-166f-4ae1-9a05-029468d94a5b", "task_name": "outpost_service_connection_state", "timestamp": "2022-09-28T12:45:16.281122"}
{"event": "Failed to get client status", "exc": "ServiceConnectionInvalid(SSHException(\"Server 'server.domain.com' not found in known_hosts\"))", "level": "warning", "logger": "authentik.outposts.tasks", "pid": 151, "task_id": "task-5d1fae0e166f4ae19a05029468d94a5b", "timestamp": "2022-09-28T12:45:17.149333"}
{"event": "Failed to get client status", "exc": "ServiceConnectionInvalid(SSHException(\"Server 'server.domain.com' not found in known_hosts\"))", "level": "warning", "logger": "authentik.outposts.tasks", "pid": 150, "task_id": "task-a19c11fd91b843dca610765e64e366ce", "timestamp": "2022-09-28T12:45:17.150311"}

additional errors in the worker log:

{"event": "Task authentik.outposts.tasks.outpost_service_connection_state[95f16633-be9b-4b2f-837a-5ddc12d0ab72] raised unexpected: DockerException('keypair must be set for SSH connections')", "exc_info": ["<class 'docker.errors.DockerException'>", "DockerException('keypair must be set for SSH connections')", "<billiard.einfo.Traceback object at 0x7f58711db370>"], "level": "error", "logger": "celery.app.trace", "timestamp": 1664369065.3033264}
{"event": "Task finished", "level": "info", "logger": "authentik.root.celery", "pid": 105, "state": "FAILURE", "task_id": "95f16633-be9b-4b2f-837a-5ddc12d0ab72", "task_name": "outpost_service_connection_state", "timestamp": "2022-09-28T12:44:25.304131"}
{"event": "Task started", "level": "info", "logger": "authentik.root.celery", "pid": 106, "task_id": "6c3558ff-3bc3-48c0-beb0-57a13c3f583b", "task_name": "outpost_service_connection_state", "timestamp": "2022-09-28T12:44:26.670166"}
{"event": "Task failure", "exc": "DockerException('keypair must be set for SSH connections')", "level": "warning", "logger": "authentik.root.celery", "pid": 106, "task_id": "task-6c3558ff3bc348c0beb057a13c3f583b", "timestamp": "2022-09-28T12:44:26.708703"}
{"event": "Task authentik.outposts.tasks.outpost_service_connection_state[6c3558ff-3bc3-48c0-beb0-57a13c3f583b] raised unexpected: DockerException('keypair must be set for SSH connections')", "exc_info": ["<class 'docker.errors.DockerException'>", "DockerException('keypair must be set for SSH connections')", "<billiard.einfo.Traceback object at 0x7f58711db460>"], "level": "error", "logger": "celery.app.trace", "timestamp": 1664369066.7094326}
{"event": "Task finished", "level": "info", "logger": "authentik.root.celery", "pid": 106, "state": "FAILURE", "task_id": "6c3558ff-3bc3-48c0-beb0-57a13c3f583b", "task_name": "outpost_service_connection_state", "timestamp": "2022-09-28T12:44:26.709652"}

[BeryJu]

hmm I'm getting similar logs but the status for the connection updates successfully

re the second error, did you set the ssh keypair on the docker connection?

[RXWatcher]

Are you getting the pem file created in /dev/shm/ ? Ultimately that would be the major issue even if the error messages related to known_hosts, etc are present. It can't do a ssh connection if the pem isnt where the config says it is.

[BeryJu]

authentik will cleanup the pem file after it's done using the client, even in failures, so the PEM file is correctly removed (in theory it should also revert the changes it made to the SSH config but that seems a bit broken currently)

[RXWatcher]

So its still an issue with the changes you implemented. It wants the knownhosts to be populated when authentik sshs. If I remove the knownhosts going to /dev/null and manually ssh in to the remote host to get it added to knownhosts then it works.

Please see these: https://github.com/docker/docker-py/issues/2398 https://github.com/docker/docker-py/issues/2398#issue-477416525

It looks like it has some ways to work around this issue.

[BeryJu]

will be fixed by #5362

[AliMickey]

Can confirm this is impacting my install as well, glad to see recent progress towards a fix soon hopefully.