SSH won't allow login or create a user if they haven't been logged in locally first

[fr3sh68]

I have the PAM module mkhomedir enabled and it works if I login from console using 'sudo login' with a user who has not logged in before but it doesn't work from SSH.

journalctl just comes back with: pam_aad(sshd:auth): Connecting to "https://login.microsoftonline.com/8c46abc7-960b> pam_aad(sshd:auth): Got response: Invalid credentials

However if I login locally journalct shows: pam_aad(login:auth): Connecting to "https://login.microsoftonline.com/8c46abc7-960b> pam_aad(login:auth): Authentication successful with user/password

The Connecting to string in both logs is the exact same but the sshd one doesn't work. I have confirmed it's the same password and not a typo by copy/pasting it.

System info: ubuntu 23.04 openssh 1:9.0p1-1ubuntu8 libpam-aad 0.4 libpam-runtime 1.5.2-5ubuntu1 libnss-aad 0.4

[denisonbarbosa]

Hey @fr3sh68, thanks for reporting your issue! I'll need more information in order to help you. Can you follow the steps listed in our troubleshooting and share the expanded logs? Remember to redact any sensitive information that is listed in the logs.

[fr3sh68]

Hi @denisonbarbosa,

I've enabled debugging for both PAM and NSS, then rebooted. FYI, I have 50 PC's with this setup and the behavior is the same on all. I took one system that a user has never logged in with and attempted to SSH, which doesn't work. I then logged in as a local admin and ran 'sudo login' and attempted to login as the same user above, which worked! Next I attempted to SSH as the newly created user and the first attempt failed, however the second attempt was successful.

All the logs with redacted sensitive information is in the attached .txt file.

aad-auth_errors_clean.txt

[ryanhulet]

Seeing the same behavior here

[wt-asw]

I also have bumped into this, is there a timeline for this issue to be resolved? :)

[denisonbarbosa]

Hey, guys! This is not a bug. SSH, as a security feature, prevents authentication if the user does not exist on the machine. When using PAM authentication, it replaces the password with a fake one to prevent password leaks through timing (when PAM is configured with a delay on fails, for example). We are still considering the best way of fixing this interaction and I'll keep you guys posted.

[ryanhulet]

If I'm not mistaken sssd, ldap, and nss can create a user on ssh login if it does not exist

[denisonbarbosa]

@ryanhulet, the workflow for SSSD and Active Directory is quite different:

With Active Directory (which is not the same as AzureAD), the machine is connected to the provider. This means that SSSD can query the domain controller for the user information required by SSH. So when SSH asks for the user information, SSSD can look somewhere else (other than its local NSS module) for that information.

Sadly, we don't have this kind of access to user information through MSAL. So when SSH asks for the user information and NSS fails to provide it, it considers the user as invalid, marks the authentication context as invalid, and then redacts the password to prevent security issues.

[wt-asw]

I found a perhaps janky work around: Basically its not that SSH doesnt work if the user hasnt logged in locally first, its that the user account hadnt been created first.

So the work around is to run a script to create the user accounts and home directories:

email_file="$1"
#
## Read the file line by line
while IFS= read -r line; do

    full_email=$(echo "$line" | tr -d '\r\n[:cntrl:]')

    # Extract the username from the email address
    username="${full_email%%@*}"
    # Set the home directory based on the username
    home_dir="/home/$username"

    # Create the user with the specified parameters
    useradd "$full_email" --home-dir "$home_dir" --shell /bin/bash --create-home

    echo "User $full_email created."

done < "$email_file"

I'm not 100% certain if this is a smart thing to do, but it has allowed me to perform some tests in docker containers. This method allowed me to authenticate using my email address without needing to log in locally first.

[gshiva]

I have the PAM module mkhomedir enabled and it works if I login from console using 'sudo login' with a user who has not logged in before but it doesn't work from SSH.

journalctl just comes back with: pam_aad(sshd:auth): Connecting to "https://login.microsoftonline.com/8c46abc7-960b> pam_aad(sshd:auth): Got response: Invalid credentials

However if I login locally journalct shows: pam_aad(login:auth): Connecting to "https://login.microsoftonline.com/8c46abc7-960b> pam_aad(login:auth): Authentication successful with user/password

The Connecting to string in both logs is the exact same but the sshd one doesn't work. I have confirmed it's the same password and not a typo by copy/pasting it.

System info: ubuntu 23.04 openssh 1:9.0p1-1ubuntu8 libpam-aad 0.4 libpam-runtime 1.5.2-5ubuntu1 libnss-aad 0.4

Thanks @fr3sh68 - for this... Hours of frustration were solved by your post. Thank you so much. Yes, the fix or warning would be nice.

[bsrishi]

More and more users are starting to face a lot. Is there a fix on this please. It's been several months.

[akulbe]

@bsrishi See this comment. This is how we work around this bug, at present.

[jh4mit]

This comment was helpful, but didn't work directly for us, since our AAD usernames ([email protected]) are almost all longer than the soft-enforced max length. I ended up writing a script to rewrite /etc/passwd and /etc/shadow directly (!). Sharing it here in case anyone else is in the same bucket:

#!/bin/bash

# Exit immediately if a command exits with a non-zero status.
set -e

# Set the default file creation mask.
umask 133

# Constants
DOMAIN="longdomain.org"
VALID_USERNAME_REGEX="^[a-z]+" # Modify as needed.
HOME_BASE="/home/$DOMAIN"
EMAIL_SUFFIX="@$DOMAIN"

# Function to display usage information
usage() {
    echo "Usage: $(basename "$0") user_principal_name"
    echo "e.g., $(basename "$0") first.last"
    exit 1
}


# Check if the input matches the required format
is_valid_format() {
    if [[ $1 =~ $VALID_USERNAME_REGEX ]]; then
        return 0
    else
        return 1
    fi
}

# Check if a user exists in the system
user_exists() {
    local username=$1
    local email="$username$EMAIL_SUFFIX"

    if getent passwd "$username" > /dev/null 2>&1; then
        echo "User '$username' already exists."
        return 0
    elif getent passwd "$email" > /dev/null 2>&1; then
        echo "User '$email' already exists."
    else
        echo "User '$username' does not exist."
        return 1
    fi
}

# Check if a home directory exists for a user
home_dir_exists() {
    local home_dir="$HOME_BASE/$1"

    if [[ -d $home_dir ]]; then
        return 0
    else
        return 1
    fi
}

# Check if a group exists
group_exists() {
    local groupname=$1

    if getent group "$groupname" > /dev/null 2>&1; then
        echo "Group '$groupname' already exists."
        return 0
    else
        return 1
    fi
}

# Compare two files
compare_files() {
    if command -v colordiff >/dev/null 2>&1; then
        colordiff -y "$@" | less -R -F -X
    else
        diff -y --color=auto "$@" | less -R -F -X
    fi
}

# Main program
username=$1
email="$username$EMAIL_SUFFIX"
home_dir="$HOME_BASE/$username"


# Check whether no arguments are provided
if [[ $# -eq 0 ]]; then
    echo "No user specified."
    usage
fi

# Check whether username is a valid format
if ! is_valid_format "$username"; then
    echo "Invalid format."
    usage
fi

# Check whether user or email already exists
if user_exists "$username"; then
    exit 1
fi

if group_exists "$username"; then
    exit 1
fi

if home_dir_exists "$username"; then
    echo "Home directory already exists, adding will not overwrite."
else
    echo "Home directory does not exist, will be created."
fi

echo "Will first create username='$username', email='$email', home_dir='$home_dir'"
read -p "Continue [y/N]? " -r
if [[ $REPLY != "y" ]]; then
    echo "Cancelled."
    exit 0
fi

echo "Creating user '$username'"
/usr/sbin/useradd "$username" --badname --home-dir "$home_dir" --user-group --shell /bin/bash --create-home

passtmp=$(mktemp)
shadowtmp=$(mktemp)

echo "Copying /etc/passwd to $passtmp"
cp /etc/passwd $passtmp

echo "Copying /etc/shadow to $shadowtmp"
cp /etc/shadow $shadowtmp

echo "Updating passwd and shadow to change $username to $email."
sed -i "s/^$username:/$email:/" $passtmp
sed -i "s/^$username:/$email:/" $shadowtmp

read -p "Press [ENTER] to view new passwd file (then q to continue)."
compare_files /etc/passwd $passtmp
read -p "Press [ENTER] to view new shadow file (then q to continue)."
compare_files /etc/shadow $shadowtmp

echo "Warning! Continuing will overwrite /etc/passwd and /etc/shadow!"
read -p "Continue [y/N]? " -r
if [[ $REPLY == "y" ]]; then
    cp --backup=numbered $passtmp /etc/passwd
    echo "/etc/passwd copied! Numbered backup saved."
    cp --backup=numbered $shadowtmp /etc/shadow
    echo "/etc/shadow copied! Numbered backup saved."
    chmod 644 /etc/passwd
    echo "/etc/passwd permissions set."
    chmod 640 /etc/shadow
    echo "/etc/shadow permissions set."
fi

rm $passtmp
rm $shadowtmp

echo "Script completed successfully."
exit 0

[blaska45]

For me login over SSH is not working at all. I've tried with 'sudo login' [email protected] and this works, but even after that with home folder etc. created, SSH with the same user [email protected] is not working. Also tried to create user [email protected] with provided scripts here by @jh4mit and @wt-asw but with SSH still no working.

login log: pam_aad(login:auth): Loading configuration from /etc/aad.conf pam_aad(login:auth): Connecting to "https://login.microsoftonline.com/xxxx", with clientID "xxxx" for user "[email protected]" pam_aad(login:auth): Authentication successful even if requiring MFA

SSH log: pam_aad(sshd:auth): Loading configuration from /etc/aad.conf pam_aad(sshd:auth): Connecting to "https://login.microsoftonline.com/xxxx", with clientID "xxxx" for user "[email protected]"

And it just keep stuck here.

In sshd_config I've tried with both KbdInteractiveAuthentication yes & no.

Can anyone help with that?

[michael-staffa]

Hey, guys! This is not a bug. SSH, as a security feature, prevents authentication if the user does not exist on the machine. When using PAM authentication, it replaces the password with a fake one to prevent password leaks through timing (when PAM is configured with a delay on fails, for example). We are still considering the best way of fixing this interaction and I'll keep you guys posted.

@denisonbarbosa Is there any progress on this? Bumping the issue.

[gshiva]

@michael-staffa thanks for bumping up the issue.

@denisonbarbosa - even a good error message saying "Cannot login user X - home folder for User X does not exist" would go a long way.