AuthorizedKeysCommand fails on Ubuntu 20.04

[zxlin]

Metadata:

• Operating System: AWS Ubuntu 20.04
• Installation method: install.sh
• AWS CLI Version: aws-cli/1.17.14 Python/3.8.2 Linux/5.4.0-1015-aws botocore/1.14.14

On Ubuntu 20.04, AuthorizedKeysCommand fails, the following is from the auth.log output AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys username SHA256:ecphulTPPp7xHnSCVkuQH2fcXemYKjT8xFftheRDz+s failed, status 22

The exact same install method works on 18.04. I'm not 100% sure what's the issue.

It looks to be related to https://github.com/widdix/aws-ec2-ssh/issues/142

My /etc/ssh/sshd_config shows on the bottom (and is the only active AuthorizedKeysCommand option):

AuthorizedKeysCommand /opt/authorized_keys_command.sh
AuthorizedKeysCommandUser nobody

My /opt/authorized_keys_command.sh is the correct one supplied by this package

The string ec2-instance-connect or eic does not appear anywhere in my /etc/ssh/sshd_config and I've triple checked reloading ssh server

Any idea why the "EC2 Instance Connect" script is being called here?

[zxlin]

Well, typing this out was helpful for me to think. I located a ec2-instance-connect package that's preinstalled on AWS Ubuntu 20.04, this package is not installed by default on 18.04. Uninstalling this package resolves this issue.

I have no idea how they're hooking into sshd to achieve this.

@michaelwittig what's the best approach here? I'm thinking:

1. Inform the user in the readme to make sure they uninstall the ec2-instance-connect package if they're on 20.04
2. auto-detecting if the package is installed and printing a conspicious message during the install process to remove the offending ec2-instance-connect package

[michaelwittig]

From the README:

Use your IAM user's public SSH key to get access via SSH to an EC2 instance running Ubuntu 16.04

So yes, 20.04 is not tested and likely is not going to work as expected yet.

[zxlin]

@michaelwittig yeah I get that, I've tested it to be working well in 18.04 (we've been using this on 18.04 for the past 2 years) and 20.04 works after removing the ec2-instance-connect package as I explained above.

Would you want to expand support for newer Ubuntu distros? Seeing that you expanded support for Amazon Linux 2 to the 2018-10-31 variant in #142, I made the assumption that you would be interested in expanding Ubuntu support, but if I am wrong, feel free to close this. Thanks!

[michaelwittig]

Yeah, sure. An update to support Ubuntu 20.04 is appreciated. And you already pointed to the ticket that explains the root cause I guess :)

[jerm]

re: "I have no idea how they're hooking into sshd to achieve this."

Because why on earth would systemd put all service configs in the same place?

One might think that /etc/systemd/ssytem/sshd.service would be where you'd find out about what sshd is meant to do (I did), but looking more carefully, that file is merely a link to /lib/systemd/system/ssh.service, where there also exists /lib/systemd/system/ssh.service.d/, in which there is this naughty file: ec2-instance-connect.conf

This adds a commandline override to only use ect-instance-connect for authing ssh sessions,

Hence uninstalling eic works.... this also works:

rm /usr/lib/systemd/system/ssh.service.d/ec2-instance-connect.conf
systemctl daemon-reload

I imagine checking for this and either asking the user to remove them or offering to do it for them would resolve the issue.

[ArtemChekunov]

apt-get remove ec2-instance-connect fixed the issue for me

[ArtemChekunov]

Hi @zxlin,

The string ec2-instance-connect or eic does not appear anywhere in my /etc/ssh/sshd_config and I've triple checked reloading ssh server

It is happening because package ec2-instance-connect install this file

cat /lib/systemd/system/ssh.service.d/ec2-instance-connect.conf
[Service]
ExecStart=
ExecStart=/usr/sbin/sshd -D -o "AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys %%u %%f" -o "AuthorizedKeysCommandUser ec2-instance-connect" $SSHD_OPTS

So, your sshd settings were overwritten